Oct 30 01:16:43.928252 ip-10-0-135-65 systemd[1]: Starting Kubernetes Kubelet... Oct 30 01:16:44.359038 ip-10-0-135-65 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 01:16:44.359038 ip-10-0-135-65 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 01:16:44.359038 ip-10-0-135-65 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 01:16:44.359038 ip-10-0-135-65 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 01:16:44.363524 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363439 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:16:44.363524 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363518 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:16:44.363524 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363522 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:16:44.363524 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363529 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:16:44.363524 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363534 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:16:44.363727 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363537 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:16:44.363727 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363540 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:16:44.363727 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363543 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:16:44.363727 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363546 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:16:44.363727 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363549 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:16:44.363727 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363552 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:16:44.363727 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363555 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:16:44.363727 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363557 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:16:44.363727 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363560 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:16:44.363727 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363563 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:16:44.363727 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363566 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:16:44.363727 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363568 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:16:44.363727 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363571 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:16:44.363727 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363579 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:16:44.363727 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363582 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:16:44.363727 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363585 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:16:44.363727 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363587 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:16:44.363727 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363590 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:16:44.363727 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363593 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:16:44.363727 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363595 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:16:44.364213 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363598 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:16:44.364213 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363601 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:16:44.364213 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363603 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:16:44.364213 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363607 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:16:44.364213 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363611 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:16:44.364213 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363614 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:16:44.364213 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363616 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:16:44.364213 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363619 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:16:44.364213 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363621 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:16:44.364213 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363624 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:16:44.364213 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363627 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:16:44.364213 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363630 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:16:44.364213 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363633 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:16:44.364213 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363636 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:16:44.364213 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363639 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:16:44.364213 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363642 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:16:44.364213 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363645 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:16:44.364213 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363648 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:16:44.364213 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363650 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:16:44.364685 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363653 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:16:44.364685 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363655 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:16:44.364685 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363658 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:16:44.364685 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363660 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:16:44.364685 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363664 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:16:44.364685 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363666 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:16:44.364685 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363669 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:16:44.364685 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363671 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:16:44.364685 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363674 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:16:44.364685 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363676 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:16:44.364685 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363679 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:16:44.364685 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363682 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:16:44.364685 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363685 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:16:44.364685 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363688 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:16:44.364685 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363690 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:16:44.364685 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363693 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:16:44.364685 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363697 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:16:44.364685 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363699 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:16:44.364685 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363702 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:16:44.365141 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363705 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:16:44.365141 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363707 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:16:44.365141 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363710 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:16:44.365141 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363713 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:16:44.365141 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363715 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:16:44.365141 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363718 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:16:44.365141 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363721 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:16:44.365141 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363724 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:16:44.365141 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363727 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:16:44.365141 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363730 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:16:44.365141 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363733 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:16:44.365141 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363736 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:16:44.365141 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363739 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:16:44.365141 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363741 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:16:44.365141 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363744 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:16:44.365141 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363747 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:16:44.365141 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363749 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:16:44.365141 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363752 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:16:44.365141 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363755 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:16:44.365623 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363758 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:16:44.365623 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363761 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:16:44.365623 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363763 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:16:44.365623 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363766 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:16:44.365623 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363770 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:16:44.365623 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363774 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:16:44.365623 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363776 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:16:44.365623 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363779 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:16:44.365623 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363782 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:16:44.365623 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363785 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:16:44.365623 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363788 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:16:44.365623 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363791 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:16:44.365623 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363793 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:16:44.365623 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363796 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:16:44.365623 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363799 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:16:44.365623 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363802 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:16:44.365623 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363805 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:16:44.365623 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363808 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:16:44.365623 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363810 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:16:44.365623 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363813 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:16:44.366104 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363816 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:16:44.366104 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363818 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:16:44.366104 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363821 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:16:44.366104 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.363823 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:16:44.366104 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364619 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:16:44.366104 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364628 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:16:44.366104 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364633 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:16:44.366104 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364637 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:16:44.366104 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364640 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:16:44.366104 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364643 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:16:44.366104 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364647 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:16:44.366104 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364649 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:16:44.366104 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364653 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:16:44.366104 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364656 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:16:44.366104 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364659 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:16:44.366104 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364661 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:16:44.366104 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364665 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:16:44.366104 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364667 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:16:44.366104 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364670 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:16:44.366553 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364673 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:16:44.366553 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364675 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:16:44.366553 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364678 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:16:44.366553 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364681 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:16:44.366553 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364683 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:16:44.366553 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364686 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:16:44.366553 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364689 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:16:44.366553 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364691 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:16:44.366553 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364694 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:16:44.366553 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364697 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:16:44.366553 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364699 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:16:44.366553 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364702 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:16:44.366553 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364705 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:16:44.366553 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364707 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:16:44.366553 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364710 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:16:44.366553 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364712 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:16:44.366553 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364715 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:16:44.366553 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364718 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:16:44.366553 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364721 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:16:44.366553 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364723 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:16:44.367029 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364726 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:16:44.367029 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364729 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:16:44.367029 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364731 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:16:44.367029 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364734 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:16:44.367029 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364736 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:16:44.367029 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364739 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:16:44.367029 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364742 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:16:44.367029 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364745 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:16:44.367029 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364747 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:16:44.367029 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364751 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:16:44.367029 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364753 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:16:44.367029 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364756 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:16:44.367029 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364759 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:16:44.367029 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364761 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:16:44.367029 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364764 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:16:44.367029 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364769 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:16:44.367029 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364772 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:16:44.367029 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364775 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:16:44.367029 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364777 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:16:44.367480 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364780 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:16:44.367480 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364783 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:16:44.367480 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364785 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:16:44.367480 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364788 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:16:44.367480 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364790 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:16:44.367480 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364793 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:16:44.367480 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364795 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:16:44.367480 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364798 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:16:44.367480 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364800 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:16:44.367480 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364803 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:16:44.367480 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364806 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:16:44.367480 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364808 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:16:44.367480 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364811 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:16:44.367480 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364814 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:16:44.367480 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364816 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:16:44.367480 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364819 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:16:44.367480 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364822 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:16:44.367480 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364824 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:16:44.367480 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364826 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:16:44.367480 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364829 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:16:44.368017 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364832 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:16:44.368017 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364835 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:16:44.368017 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364840 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:16:44.368017 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364842 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:16:44.368017 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364845 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:16:44.368017 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364848 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:16:44.368017 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364850 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:16:44.368017 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364853 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:16:44.368017 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364855 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:16:44.368017 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364858 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:16:44.368017 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364860 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:16:44.368017 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364863 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:16:44.368017 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364865 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:16:44.368017 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364868 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:16:44.368017 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364870 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:16:44.368017 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364873 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:16:44.368017 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364877 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:16:44.368017 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364880 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:16:44.368017 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364882 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:16:44.368017 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364885 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:16:44.368487 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364888 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:16:44.368487 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364890 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:16:44.368487 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364893 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:16:44.368487 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364896 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:16:44.368487 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364899 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:16:44.368487 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364901 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:16:44.368487 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364903 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:16:44.368487 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364906 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:16:44.368487 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364909 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:16:44.368487 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364912 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:16:44.368487 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364915 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:16:44.368487 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.364917 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:16:44.368487 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366086 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:16:44.368487 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366095 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:16:44.368487 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366101 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:16:44.368487 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366105 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:16:44.368487 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366108 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:16:44.368487 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366111 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:16:44.368487 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366115 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:16:44.368487 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366118 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:16:44.368966 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366120 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:16:44.368966 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366123 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:16:44.368966 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366126 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:16:44.368966 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366128 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:16:44.368966 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366131 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:16:44.368966 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366134 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:16:44.368966 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366136 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:16:44.368966 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366139 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:16:44.368966 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366142 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:16:44.368966 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366145 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:16:44.368966 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366148 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:16:44.368966 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366151 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:16:44.368966 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366153 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:16:44.368966 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366156 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:16:44.368966 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366159 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:16:44.368966 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366161 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:16:44.368966 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366164 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:16:44.368966 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366166 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:16:44.368966 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366169 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:16:44.368966 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366171 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:16:44.369435 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366174 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:16:44.369435 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366177 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:16:44.369435 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366179 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:16:44.369435 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366182 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:16:44.369435 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366185 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:16:44.369435 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366188 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:16:44.369435 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366192 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:16:44.369435 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366194 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:16:44.369435 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366197 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:16:44.369435 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366200 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:16:44.369435 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366202 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:16:44.369435 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366205 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:16:44.369435 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366207 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:16:44.369435 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366210 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:16:44.369435 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366212 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:16:44.369435 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366215 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:16:44.369435 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366217 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:16:44.369435 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366220 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:16:44.369435 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366222 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:16:44.369877 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366225 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:16:44.369877 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366228 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:16:44.369877 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366231 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:16:44.369877 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366233 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:16:44.369877 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366236 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:16:44.369877 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366239 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:16:44.369877 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366241 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:16:44.369877 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366244 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:16:44.369877 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366246 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:16:44.369877 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366249 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:16:44.369877 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366251 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:16:44.369877 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366254 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:16:44.369877 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366256 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:16:44.369877 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366259 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:16:44.369877 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366264 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:16:44.369877 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366267 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:16:44.369877 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366271 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:16:44.369877 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366273 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:16:44.369877 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366276 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:16:44.370333 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366286 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:16:44.370333 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366289 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:16:44.370333 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366292 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:16:44.370333 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366294 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:16:44.370333 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366297 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:16:44.370333 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366301 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:16:44.370333 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366305 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:16:44.370333 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366309 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:16:44.370333 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366311 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:16:44.370333 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366314 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:16:44.370333 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366317 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:16:44.370333 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366319 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:16:44.370333 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366322 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:16:44.370333 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366325 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:16:44.370333 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366327 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:16:44.370333 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366330 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:16:44.370333 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366332 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:16:44.370333 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366335 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:16:44.370333 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366338 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:16:44.370772 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366341 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:16:44.370772 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366343 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:16:44.370772 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366346 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:16:44.370772 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366348 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:16:44.370772 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366351 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:16:44.370772 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366353 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:16:44.370772 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366356 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:16:44.370772 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366358 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:16:44.370772 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366361 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:16:44.370772 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366364 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:16:44.370772 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366366 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:16:44.370772 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366369 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:16:44.370772 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366371 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:16:44.370772 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366374 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:16:44.370772 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366377 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:16:44.370772 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366381 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:16:44.370772 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366385 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:16:44.370772 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366389 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:16:44.370772 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366392 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:16:44.371222 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366396 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:16:44.371222 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.366399 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:16:44.371222 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367375 2573 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 01:16:44.371222 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367390 2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 01:16:44.371222 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367396 2573 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 01:16:44.371222 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367400 2573 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 01:16:44.371222 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367405 2573 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 01:16:44.371222 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367409 2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 01:16:44.371222 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367413 2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 01:16:44.371222 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367417 2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 01:16:44.371222 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367420 2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 01:16:44.371222 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367424 2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 01:16:44.371222 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367427 2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 01:16:44.371222 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367431 2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 01:16:44.371222 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367434 2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 01:16:44.371222 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367437 2573 flags.go:64] FLAG: --cgroup-root="" Oct 30 01:16:44.371222 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367440 2573 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 01:16:44.371222 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367443 2573 flags.go:64] FLAG: --client-ca-file="" Oct 30 01:16:44.371222 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367447 2573 flags.go:64] FLAG: --cloud-provider="external" Oct 30 01:16:44.371222 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367450 2573 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 01:16:44.371222 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367455 2573 flags.go:64] FLAG: --cluster-domain="" Oct 30 01:16:44.371222 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367458 2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 01:16:44.371222 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367461 2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 01:16:44.371757 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367464 2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 01:16:44.371757 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367471 2573 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 01:16:44.371757 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367475 2573 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 01:16:44.371757 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367478 2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 01:16:44.371757 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367481 2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 01:16:44.371757 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367485 2573 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 01:16:44.371757 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367489 2573 flags.go:64] FLAG: --contention-profiling="false" Oct 30 01:16:44.371757 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367492 2573 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 01:16:44.371757 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367495 2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 01:16:44.371757 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367499 2573 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 01:16:44.371757 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367502 2573 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 01:16:44.371757 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367506 2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 01:16:44.371757 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367509 2573 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 01:16:44.371757 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367512 2573 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 01:16:44.371757 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367515 2573 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 01:16:44.371757 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367518 2573 flags.go:64] FLAG: --enable-server="true" Oct 30 01:16:44.371757 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367521 2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 01:16:44.371757 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367525 2573 flags.go:64] FLAG: --event-burst="100" Oct 30 01:16:44.371757 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367528 2573 flags.go:64] FLAG: --event-qps="50" Oct 30 01:16:44.371757 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367534 2573 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 01:16:44.371757 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367537 2573 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 01:16:44.371757 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367540 2573 flags.go:64] FLAG: --eviction-hard="" Oct 30 01:16:44.371757 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367545 2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 01:16:44.371757 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367548 2573 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 01:16:44.372380 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367551 2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 01:16:44.372380 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367555 2573 flags.go:64] FLAG: --eviction-soft="" Oct 30 01:16:44.372380 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367558 2573 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 01:16:44.372380 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367561 2573 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 01:16:44.372380 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367564 2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 01:16:44.372380 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367567 2573 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 01:16:44.372380 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367570 2573 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 01:16:44.372380 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367573 2573 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 01:16:44.372380 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367576 2573 flags.go:64] FLAG: --feature-gates="" Oct 30 01:16:44.372380 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367581 2573 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 01:16:44.372380 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367585 2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 01:16:44.372380 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367588 2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 01:16:44.372380 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367592 2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 01:16:44.372380 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367595 2573 flags.go:64] FLAG: --healthz-port="10248" Oct 30 01:16:44.372380 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367599 2573 flags.go:64] FLAG: --help="false" Oct 30 01:16:44.372380 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367602 2573 flags.go:64] FLAG: --hostname-override="ip-10-0-135-65.ec2.internal" Oct 30 01:16:44.372380 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367606 2573 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 01:16:44.372380 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367609 2573 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 01:16:44.372380 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367612 2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 01:16:44.372380 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367615 2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 01:16:44.372380 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367619 2573 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 01:16:44.372380 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367622 2573 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 01:16:44.372380 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367625 2573 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 01:16:44.372942 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367628 2573 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 01:16:44.372942 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367631 2573 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 01:16:44.372942 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367634 2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 01:16:44.372942 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367637 2573 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 01:16:44.372942 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367641 2573 flags.go:64] FLAG: --kube-reserved="" Oct 30 01:16:44.372942 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367644 2573 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 01:16:44.372942 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367647 2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 01:16:44.372942 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367651 2573 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 01:16:44.372942 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367654 2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 01:16:44.372942 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367657 2573 flags.go:64] FLAG: --lock-file="" Oct 30 01:16:44.372942 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367660 2573 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 01:16:44.372942 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367663 2573 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 01:16:44.372942 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367666 2573 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 01:16:44.372942 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367672 2573 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 01:16:44.372942 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367675 2573 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 01:16:44.372942 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367679 2573 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 01:16:44.372942 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367682 2573 flags.go:64] FLAG: --logging-format="text" Oct 30 01:16:44.372942 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367685 2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 01:16:44.372942 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367690 2573 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 01:16:44.372942 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367693 2573 flags.go:64] FLAG: --manifest-url="" Oct 30 01:16:44.372942 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367696 2573 flags.go:64] FLAG: --manifest-url-header="" Oct 30 01:16:44.372942 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367701 2573 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 01:16:44.372942 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367704 2573 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 01:16:44.372942 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367708 2573 flags.go:64] FLAG: --max-pods="110" Oct 30 01:16:44.372942 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367712 2573 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 01:16:44.373522 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367715 2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 01:16:44.373522 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367719 2573 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 01:16:44.373522 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367722 2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 01:16:44.373522 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367725 2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 01:16:44.373522 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367728 2573 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 01:16:44.373522 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367731 2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 01:16:44.373522 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367739 2573 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 01:16:44.373522 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367742 2573 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 01:16:44.373522 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367745 2573 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 01:16:44.373522 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367748 2573 flags.go:64] FLAG: --pod-cidr="" Oct 30 01:16:44.373522 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367751 2573 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 01:16:44.373522 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367755 2573 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 01:16:44.373522 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367759 2573 flags.go:64] FLAG: --pods-per-core="0" Oct 30 01:16:44.373522 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367762 2573 flags.go:64] FLAG: --port="10250" Oct 30 01:16:44.373522 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367765 2573 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 01:16:44.373522 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367768 2573 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00c04657cfc51c167" Oct 30 01:16:44.373522 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367771 2573 flags.go:64] FLAG: --qos-reserved="" Oct 30 01:16:44.373522 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367774 2573 flags.go:64] FLAG: --read-only-port="10255" Oct 30 01:16:44.373522 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367777 2573 flags.go:64] FLAG: --register-node="true" Oct 30 01:16:44.373522 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367780 2573 flags.go:64] FLAG: --register-with-taints="" Oct 30 01:16:44.373522 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367785 2573 flags.go:64] FLAG: --registry-burst="10" Oct 30 01:16:44.373522 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367788 2573 flags.go:64] FLAG: --registry-qps="5" Oct 30 01:16:44.373522 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367791 2573 flags.go:64] FLAG: --reserved-cpus="" Oct 30 01:16:44.373522 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367794 2573 flags.go:64] FLAG: --reserved-memory="" Oct 30 01:16:44.373522 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367798 2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 01:16:44.373522 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367803 2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 01:16:44.374138 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367806 2573 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 01:16:44.374138 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367809 2573 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 01:16:44.374138 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367812 2573 flags.go:64] FLAG: --runonce="false" Oct 30 01:16:44.374138 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367815 2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 01:16:44.374138 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367818 2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 01:16:44.374138 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367821 2573 flags.go:64] FLAG: --seccomp-default="false" Oct 30 01:16:44.374138 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367825 2573 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 01:16:44.374138 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367828 2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 01:16:44.374138 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367831 2573 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 01:16:44.374138 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367834 2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 01:16:44.374138 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367838 2573 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 01:16:44.374138 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367841 2573 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 01:16:44.374138 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367844 2573 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 01:16:44.374138 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367847 2573 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 01:16:44.374138 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367850 2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 01:16:44.374138 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367853 2573 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 01:16:44.374138 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367863 2573 flags.go:64] FLAG: --system-cgroups="" Oct 30 01:16:44.374138 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367869 2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 01:16:44.374138 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367874 2573 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 01:16:44.374138 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367877 2573 flags.go:64] FLAG: --tls-cert-file="" Oct 30 01:16:44.374138 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367880 2573 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 01:16:44.374138 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367885 2573 flags.go:64] FLAG: --tls-min-version="" Oct 30 01:16:44.374138 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367888 2573 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 01:16:44.374138 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367891 2573 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 01:16:44.374138 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367894 2573 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 01:16:44.374720 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367897 2573 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 01:16:44.374720 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367900 2573 flags.go:64] FLAG: --v="2" Oct 30 01:16:44.374720 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367904 2573 flags.go:64] FLAG: --version="false" Oct 30 01:16:44.374720 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367909 2573 flags.go:64] FLAG: --vmodule="" Oct 30 01:16:44.374720 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367913 2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 01:16:44.374720 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.367916 2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 01:16:44.374720 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368076 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:16:44.374720 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368081 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:16:44.374720 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368084 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:16:44.374720 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368087 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:16:44.374720 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368090 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:16:44.374720 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368093 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:16:44.374720 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368096 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:16:44.374720 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368098 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:16:44.374720 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368107 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:16:44.374720 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368111 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:16:44.374720 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368114 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:16:44.374720 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368117 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:16:44.374720 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368119 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:16:44.374720 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368122 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:16:44.374720 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368124 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:16:44.375231 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368127 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:16:44.375231 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368130 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:16:44.375231 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368133 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:16:44.375231 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368137 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:16:44.375231 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368139 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:16:44.375231 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368144 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:16:44.375231 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368148 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:16:44.375231 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368151 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:16:44.375231 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368154 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:16:44.375231 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368157 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:16:44.375231 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368160 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:16:44.375231 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368163 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:16:44.375231 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368165 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:16:44.375231 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368168 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:16:44.375231 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368171 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:16:44.375231 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368174 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:16:44.375231 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368176 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:16:44.375231 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368180 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:16:44.375231 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368182 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:16:44.375684 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368185 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:16:44.375684 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368188 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:16:44.375684 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368190 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:16:44.375684 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368193 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:16:44.375684 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368196 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:16:44.375684 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368198 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:16:44.375684 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368201 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:16:44.375684 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368204 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:16:44.375684 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368207 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:16:44.375684 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368210 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:16:44.375684 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368212 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:16:44.375684 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368215 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:16:44.375684 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368218 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:16:44.375684 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368220 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:16:44.375684 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368223 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:16:44.375684 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368226 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:16:44.375684 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368230 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:16:44.375684 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368232 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:16:44.375684 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368235 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:16:44.375684 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368238 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:16:44.376242 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368240 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:16:44.376242 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368243 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:16:44.376242 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368246 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:16:44.376242 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368248 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:16:44.376242 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368251 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:16:44.376242 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368254 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:16:44.376242 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368257 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:16:44.376242 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368260 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:16:44.376242 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368262 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:16:44.376242 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368265 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:16:44.376242 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368268 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:16:44.376242 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368270 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:16:44.376242 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368273 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:16:44.376242 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368276 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:16:44.376242 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368278 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:16:44.376242 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368281 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:16:44.376242 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368283 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:16:44.376242 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368286 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:16:44.376242 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368289 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:16:44.376731 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368292 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:16:44.376731 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368295 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:16:44.376731 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368298 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:16:44.376731 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368300 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:16:44.376731 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368303 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:16:44.376731 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368307 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:16:44.376731 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368310 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:16:44.376731 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368313 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:16:44.376731 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368316 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:16:44.376731 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368320 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:16:44.376731 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368323 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:16:44.376731 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368325 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:16:44.376731 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368328 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:16:44.376731 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368331 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:16:44.376731 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368333 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:16:44.376731 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368336 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:16:44.376731 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368339 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:16:44.376731 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368342 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:16:44.376731 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368344 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:16:44.376731 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368347 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:16:44.377209 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368350 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:16:44.377209 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368352 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:16:44.377209 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368355 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:16:44.377209 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368358 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:16:44.377209 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368360 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:16:44.377209 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368363 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:16:44.377209 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368365 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:16:44.377209 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368368 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:16:44.377209 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368371 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:16:44.377209 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368373 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:16:44.377209 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368376 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:16:44.377209 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368379 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:16:44.377209 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368381 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:16:44.377209 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.368407 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 01:16:44.377209 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368449 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:16:44.377209 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368453 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:16:44.377585 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368456 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:16:44.377585 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368459 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:16:44.377585 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368461 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:16:44.377585 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368464 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:16:44.377585 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368467 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:16:44.377585 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368469 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:16:44.377585 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368473 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:16:44.377585 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368476 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:16:44.377585 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368479 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:16:44.377585 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368482 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:16:44.377585 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368484 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:16:44.377585 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368487 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:16:44.377585 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368490 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:16:44.377585 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368493 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:16:44.377585 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368496 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:16:44.377585 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368499 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:16:44.377585 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368502 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:16:44.377585 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368504 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:16:44.377585 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368507 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:16:44.377585 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368510 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:16:44.378086 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368512 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:16:44.378086 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368515 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:16:44.378086 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368518 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:16:44.378086 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368520 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:16:44.378086 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368523 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:16:44.378086 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368525 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:16:44.378086 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368528 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:16:44.378086 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368531 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:16:44.378086 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368533 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:16:44.378086 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368536 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:16:44.378086 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368538 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:16:44.378086 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368541 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:16:44.378086 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368544 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:16:44.378086 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368548 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:16:44.378086 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368551 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:16:44.378086 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368554 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:16:44.378086 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368556 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:16:44.378086 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368559 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:16:44.378086 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368563 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:16:44.378520 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368567 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:16:44.378520 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368572 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:16:44.378520 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368575 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:16:44.378520 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368578 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:16:44.378520 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368580 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:16:44.378520 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368583 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:16:44.378520 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368586 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:16:44.378520 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368588 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:16:44.378520 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368591 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:16:44.378520 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368593 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:16:44.378520 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368596 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:16:44.378520 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368598 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:16:44.378520 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368601 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:16:44.378520 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368604 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:16:44.378520 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368606 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:16:44.378520 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368609 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:16:44.378520 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368612 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:16:44.378520 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368614 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:16:44.378520 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368617 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:16:44.379000 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368619 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:16:44.379000 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368622 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:16:44.379000 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368625 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:16:44.379000 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368627 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:16:44.379000 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368630 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:16:44.379000 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368632 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:16:44.379000 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368635 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:16:44.379000 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368639 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:16:44.379000 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368642 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:16:44.379000 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368645 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:16:44.379000 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368647 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:16:44.379000 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368650 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:16:44.379000 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368655 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:16:44.379000 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368657 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:16:44.379000 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368660 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:16:44.379000 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368662 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:16:44.379000 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368665 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:16:44.379000 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368668 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:16:44.379000 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368670 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:16:44.379000 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368673 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:16:44.379476 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368675 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:16:44.379476 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368678 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:16:44.379476 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368681 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:16:44.379476 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368684 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:16:44.379476 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368687 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:16:44.379476 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368689 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:16:44.379476 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368692 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:16:44.379476 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368694 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:16:44.379476 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368697 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:16:44.379476 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368699 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:16:44.379476 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368702 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:16:44.379476 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368704 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:16:44.379476 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368707 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:16:44.379476 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368710 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:16:44.379476 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368712 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:16:44.379476 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368715 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:16:44.379476 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368718 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:16:44.379476 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368720 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:16:44.379476 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368723 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:16:44.379922 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368729 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:16:44.379922 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368733 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:16:44.379922 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368736 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:16:44.379922 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368739 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:16:44.379922 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368741 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:16:44.379922 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368746 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:16:44.379922 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.368749 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:16:44.379922 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.375405 2573 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 01:16:44.379922 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.375419 2573 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 01:16:44.379922 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.375435 2573 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 01:16:44.379922 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.375439 2573 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 01:16:44.379922 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375497 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:16:44.379922 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375502 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:16:44.379922 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375506 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:16:44.379922 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375509 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:16:44.379922 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375511 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:16:44.379922 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375514 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:16:44.379922 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375517 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:16:44.379922 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375520 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:16:44.380385 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375523 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:16:44.380385 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375526 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:16:44.380385 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375528 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:16:44.380385 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375531 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:16:44.380385 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375534 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:16:44.380385 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375537 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:16:44.380385 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375539 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:16:44.380385 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375542 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:16:44.380385 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375546 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:16:44.380385 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375550 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:16:44.380385 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375554 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:16:44.380385 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375558 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:16:44.380385 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375562 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:16:44.380385 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375565 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:16:44.380385 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375568 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:16:44.380385 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375571 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:16:44.380385 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375573 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:16:44.380385 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375576 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:16:44.380385 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375579 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:16:44.380830 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375581 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:16:44.380830 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375584 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:16:44.380830 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375588 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:16:44.380830 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375591 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:16:44.380830 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375594 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:16:44.380830 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375597 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:16:44.380830 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375600 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:16:44.380830 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375602 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:16:44.380830 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375605 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:16:44.380830 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375608 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:16:44.380830 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375611 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:16:44.380830 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375613 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:16:44.380830 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375616 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:16:44.380830 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375619 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:16:44.380830 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375622 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:16:44.380830 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375624 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:16:44.380830 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375627 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:16:44.380830 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375630 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:16:44.380830 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375632 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:16:44.380830 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375635 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:16:44.381314 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375638 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:16:44.381314 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375640 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:16:44.381314 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375643 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:16:44.381314 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375646 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:16:44.381314 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375649 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:16:44.381314 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375651 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:16:44.381314 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375654 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:16:44.381314 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375657 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:16:44.381314 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375660 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:16:44.381314 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375663 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:16:44.381314 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375665 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:16:44.381314 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375668 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:16:44.381314 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375670 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:16:44.381314 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375673 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:16:44.381314 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375677 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:16:44.381314 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375680 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:16:44.381314 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375682 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:16:44.381314 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375685 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:16:44.381314 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375688 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:16:44.381751 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375691 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:16:44.381751 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375694 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:16:44.381751 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375696 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:16:44.381751 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375699 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:16:44.381751 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375702 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:16:44.381751 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375705 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:16:44.381751 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375707 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:16:44.381751 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375710 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:16:44.381751 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375712 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:16:44.381751 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375715 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:16:44.381751 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375718 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:16:44.381751 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375720 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:16:44.381751 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375723 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:16:44.381751 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375726 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:16:44.381751 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375728 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:16:44.381751 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375731 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:16:44.381751 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375734 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:16:44.381751 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375737 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:16:44.381751 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375739 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:16:44.381751 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375742 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:16:44.382267 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375745 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:16:44.382267 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375748 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:16:44.382267 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375750 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:16:44.382267 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375753 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:16:44.382267 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375756 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:16:44.382267 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375758 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:16:44.382267 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375761 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:16:44.382267 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375764 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:16:44.382267 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375766 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:16:44.382267 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375769 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:16:44.382267 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375772 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:16:44.382267 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375774 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:16:44.382267 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375777 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:16:44.382267 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375779 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:16:44.382267 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375782 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:16:44.382267 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375785 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:16:44.382267 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375787 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:16:44.382267 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375790 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:16:44.382267 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375792 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:16:44.382699 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375795 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:16:44.382699 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.375821 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 01:16:44.382699 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375983 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:16:44.382699 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375989 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:16:44.382699 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375992 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:16:44.382699 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375994 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:16:44.382699 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.375997 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:16:44.382699 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376000 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:16:44.382699 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376002 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:16:44.382699 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376005 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:16:44.382699 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376008 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:16:44.382699 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376011 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:16:44.382699 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376013 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:16:44.382699 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376016 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:16:44.382699 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376019 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:16:44.382699 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376022 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:16:44.383198 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376025 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:16:44.383198 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376027 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:16:44.383198 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376030 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:16:44.383198 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376033 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:16:44.383198 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376035 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:16:44.383198 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376038 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:16:44.383198 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376041 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:16:44.383198 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376044 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:16:44.383198 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376046 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:16:44.383198 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376049 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:16:44.383198 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376051 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:16:44.383198 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376055 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:16:44.383198 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376059 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:16:44.383198 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376062 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:16:44.383198 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376065 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:16:44.383198 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376067 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:16:44.383198 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376070 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:16:44.383198 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376073 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:16:44.383198 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376075 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:16:44.383667 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376078 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:16:44.383667 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376080 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:16:44.383667 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376084 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:16:44.383667 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376086 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:16:44.383667 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376089 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:16:44.383667 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376091 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:16:44.383667 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376094 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:16:44.383667 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376096 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:16:44.383667 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376099 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:16:44.383667 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376102 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:16:44.383667 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376105 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:16:44.383667 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376107 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:16:44.383667 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376110 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:16:44.383667 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376113 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:16:44.383667 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376116 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:16:44.383667 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376118 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:16:44.383667 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376121 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:16:44.383667 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376123 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:16:44.383667 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376126 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:16:44.384130 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376129 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:16:44.384130 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376131 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:16:44.384130 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376134 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:16:44.384130 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376136 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:16:44.384130 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376139 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:16:44.384130 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376143 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:16:44.384130 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376147 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:16:44.384130 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376151 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:16:44.384130 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376154 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:16:44.384130 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376156 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:16:44.384130 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376159 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:16:44.384130 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376162 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:16:44.384130 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376164 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:16:44.384130 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376168 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:16:44.384130 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376170 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:16:44.384130 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376173 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:16:44.384130 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376175 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:16:44.384130 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376178 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:16:44.384130 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376181 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:16:44.384576 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376183 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:16:44.384576 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376186 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:16:44.384576 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376189 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:16:44.384576 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376191 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:16:44.384576 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376194 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:16:44.384576 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376196 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:16:44.384576 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376199 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:16:44.384576 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376202 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:16:44.384576 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376205 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:16:44.384576 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376208 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:16:44.384576 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376210 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:16:44.384576 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376213 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:16:44.384576 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376215 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:16:44.384576 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376218 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:16:44.384576 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376221 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:16:44.384576 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376223 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:16:44.384576 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376226 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:16:44.384576 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376228 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:16:44.384576 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376231 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:16:44.385019 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376234 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:16:44.385019 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376236 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:16:44.385019 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376239 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:16:44.385019 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376241 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:16:44.385019 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376244 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:16:44.385019 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376247 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:16:44.385019 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376250 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:16:44.385019 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376252 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:16:44.385019 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376255 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:16:44.385019 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376258 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:16:44.385019 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376260 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:16:44.385019 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376263 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:16:44.385019 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376265 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:16:44.385019 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376268 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:16:44.385019 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376270 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:16:44.385019 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376273 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:16:44.385019 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.376296 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 01:16:44.385414 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376329 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:16:44.385414 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376333 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:16:44.385414 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376336 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:16:44.385414 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376338 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:16:44.385414 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376341 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:16:44.385414 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376345 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:16:44.385414 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376349 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:16:44.385414 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376353 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:16:44.385414 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376356 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:16:44.385414 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376358 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:16:44.385414 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376361 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:16:44.385414 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376364 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:16:44.385414 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376366 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:16:44.385414 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376369 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:16:44.385414 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376371 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:16:44.385414 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376374 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:16:44.385414 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376376 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:16:44.385414 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376379 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:16:44.385414 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376382 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:16:44.385838 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376384 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:16:44.385838 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376387 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:16:44.385838 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376389 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:16:44.385838 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376392 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:16:44.385838 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376395 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:16:44.385838 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376397 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:16:44.385838 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376400 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:16:44.385838 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376402 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:16:44.385838 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376405 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:16:44.385838 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376407 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:16:44.385838 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376410 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:16:44.385838 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376412 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:16:44.385838 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376415 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:16:44.385838 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376417 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:16:44.385838 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376420 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:16:44.385838 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376422 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:16:44.385838 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376425 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:16:44.385838 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376427 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:16:44.385838 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376431 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:16:44.386429 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376435 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:16:44.386429 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376438 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:16:44.386429 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376441 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:16:44.386429 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376443 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:16:44.386429 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376446 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:16:44.386429 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376448 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:16:44.386429 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376450 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:16:44.386429 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376454 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:16:44.386429 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376456 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:16:44.386429 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376459 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:16:44.386429 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376461 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:16:44.386429 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376464 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:16:44.386429 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376466 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:16:44.386429 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376469 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:16:44.386429 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376471 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:16:44.386429 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376474 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:16:44.386429 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376477 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:16:44.386429 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376481 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:16:44.386429 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376484 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:16:44.386429 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376486 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:16:44.387184 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376489 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:16:44.387184 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376492 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:16:44.387184 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376494 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:16:44.387184 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376496 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:16:44.387184 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376499 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:16:44.387184 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376502 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:16:44.387184 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376504 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:16:44.387184 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376507 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:16:44.387184 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376509 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:16:44.387184 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376512 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:16:44.387184 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376514 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:16:44.387184 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376517 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:16:44.387184 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376519 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:16:44.387184 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376522 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:16:44.387184 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376525 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:16:44.387184 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376528 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:16:44.387184 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376530 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:16:44.387184 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376533 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:16:44.387184 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376535 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:16:44.387184 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376538 2573 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:16:44.387742 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376540 2573 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:16:44.387742 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376545 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:16:44.387742 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376547 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:16:44.387742 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376550 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:16:44.387742 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376553 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:16:44.387742 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376555 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:16:44.387742 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376558 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:16:44.387742 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376561 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:16:44.387742 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376563 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:16:44.387742 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376567 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:16:44.387742 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376570 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:16:44.387742 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376572 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:16:44.387742 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376575 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:16:44.387742 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376577 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:16:44.387742 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376580 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:16:44.387742 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376582 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:16:44.387742 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376585 2573 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:16:44.387742 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376588 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:16:44.387742 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376591 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:16:44.387742 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376593 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:16:44.387742 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376596 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:16:44.388450 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376598 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:16:44.388450 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376601 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:16:44.388450 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376604 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:16:44.388450 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376606 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:16:44.388450 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376610 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:16:44.388450 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376614 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:16:44.388450 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:44.376616 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:16:44.388450 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.376664 2573 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 01:16:44.388450 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.378533 2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 01:16:44.388450 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.379471 2573 server.go:1014] "Starting client certificate rotation" Oct 30 01:16:44.388450 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.379572 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 01:16:44.388450 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.379605 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 01:16:44.388450 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.382005 2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 01:16:44.388450 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.383199 2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 01:16:44.393747 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.393728 2573 log.go:25] "Validated CRI v1 runtime API" Oct 30 01:16:44.396677 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.396660 2573 log.go:25] "Validated CRI v1 image API" Oct 30 01:16:44.397451 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.397424 2573 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 01:16:44.402010 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.401989 2573 fs.go:135] Filesystem UUIDs: map[3ccc9cf7-84c2-442d-8b45-3ea4bc2c9a64:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 aab3d7db-4c1f-4cea-b5d4-f0a784d68598:/dev/nvme0n1p3] Oct 30 01:16:44.402094 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.402009 2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 01:16:44.410658 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.410523 2573 manager.go:217] Machine: {Timestamp:2026-10-30 01:16:44.405726606 +0000 UTC m=+0.355298718 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099050 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec247f29ced8496ed48af6e583013cf9 SystemUUID:ec247f29-ced8-496e-d48a-f6e583013cf9 BootID:9720e4d8-b2b4-4456-be40-c0ce7217ea1f Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c1:68:de:53 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c1:68:de:53 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:46:27:7e:4f:14:bf Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 01:16:44.410658 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.410650 2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 01:16:44.410822 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.410802 2573 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 01:16:44.413541 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.413519 2573 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 01:16:44.413681 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.413542 2573 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-65.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 01:16:44.413733 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.413683 2573 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 01:16:44.413733 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.413692 2573 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 01:16:44.413733 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.413711 2573 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 01:16:44.413733 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.413726 2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 01:16:44.414456 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.414446 2573 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 01:16:44.418068 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.418055 2573 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 01:16:44.418177 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.418168 2573 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 01:16:44.418856 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.418847 2573 kubelet.go:493] "Attempting to sync node with API server" Oct 30 01:16:44.418894 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.418860 2573 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 01:16:44.418894 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.418873 2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 01:16:44.418894 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.418883 2573 kubelet.go:405] "Adding apiserver pod source" Oct 30 01:16:44.418894 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.418892 2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 01:16:44.419211 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.419200 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 01:16:44.421337 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.421318 2573 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 01:16:44.425370 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.425356 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 01:16:44.427229 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.427212 2573 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 01:16:44.427316 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.427243 2573 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 01:16:44.427440 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.427426 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 01:16:44.427491 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.427449 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 01:16:44.427491 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.427456 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 01:16:44.427491 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.427462 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 01:16:44.427491 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.427468 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 01:16:44.427491 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.427475 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 01:16:44.427965 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.427955 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 01:16:44.427965 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.427965 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 01:16:44.428027 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.427973 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 01:16:44.428027 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.427979 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 01:16:44.428027 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.427988 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 01:16:44.428027 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.427997 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 01:16:44.428732 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.428723 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 01:16:44.428732 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.428732 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 01:16:44.432247 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.432236 2573 server.go:1263] "Started kubelet" Oct 30 01:16:44.432354 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.432327 2573 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 01:16:44.432560 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.432516 2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 01:16:44.432632 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.432573 2573 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 01:16:44.433734 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.433710 2573 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 01:16:44.434207 ip-10-0-135-65 systemd[1]: Started Kubernetes Kubelet. Oct 30 01:16:44.438813 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:44.438790 2573 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 01:16:44.439376 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.439359 2573 server.go:317] "Adding debug handlers to kubelet server" Oct 30 01:16:44.440526 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.440506 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 01:16:44.441002 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.440987 2573 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 01:16:44.441168 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:44.441112 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-65.ec2.internal\" not found" Oct 30 01:16:44.441618 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.441604 2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 01:16:44.441765 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.441712 2573 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 01:16:44.441765 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.441763 2573 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 01:16:44.441877 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.441733 2573 factory.go:55] Registering systemd factory Oct 30 01:16:44.441877 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.441811 2573 factory.go:223] Registration of the systemd container factory successfully Oct 30 01:16:44.441877 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.441860 2573 reconstruct.go:97] "Volume reconstruction finished" Oct 30 01:16:44.441877 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.441871 2573 reconciler.go:29] "Reconciler: start to sync state" Oct 30 01:16:44.442076 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.442064 2573 factory.go:153] Registering CRI-O factory Oct 30 01:16:44.442136 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.442079 2573 factory.go:223] Registration of the crio container factory successfully Oct 30 01:16:44.442136 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.442127 2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 01:16:44.442230 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.442145 2573 factory.go:103] Registering Raw factory Oct 30 01:16:44.442230 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.442160 2573 manager.go:1199] Started watching for new ooms in manager Oct 30 01:16:44.443227 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.443213 2573 manager.go:319] Starting recovery of all containers Oct 30 01:16:44.453271 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.453254 2573 manager.go:324] Recovery completed Oct 30 01:16:44.457324 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.457310 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 01:16:44.458986 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.458970 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-65.ec2.internal" event="NodeHasSufficientMemory" Oct 30 01:16:44.459046 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.459000 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-65.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 01:16:44.459046 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.459010 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-65.ec2.internal" event="NodeHasSufficientPID" Oct 30 01:16:44.459540 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.459528 2573 cpu_manager.go:226] "Starting" policy="none" Oct 30 01:16:44.459584 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.459539 2573 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 01:16:44.459584 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.459564 2573 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 01:16:44.459584 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.459568 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 01:16:44.460210 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.460193 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-65.ec2.internal" event="NodeHasSufficientMemory" Oct 30 01:16:44.460279 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.460221 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-65.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 01:16:44.460279 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.460230 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-65.ec2.internal" event="NodeHasSufficientPID" Oct 30 01:16:44.461734 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.461722 2573 policy_none.go:50] "Start" Oct 30 01:16:44.461768 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.461743 2573 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 01:16:44.461768 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.461754 2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 01:16:44.463348 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.463333 2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 01:16:44.463410 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.463369 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5r8rf" Oct 30 01:16:44.463657 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.463645 2573 policy_none.go:44] "Start" Oct 30 01:16:44.464541 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:44.464516 2573 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-65.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 01:16:44.464541 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.464507 2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-65.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 01:16:44.465747 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.465730 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5r8rf" Oct 30 01:16:44.473536 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:44.464518 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-65.ec2.internal.18e329c566b7d861 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-65.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-65.ec2.internal,},FirstTimestamp:2026-10-30 01:16:44.432210017 +0000 UTC m=+0.381782125,LastTimestamp:2026-10-30 01:16:44.432210017 +0000 UTC m=+0.381782125,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-65.ec2.internal,}" Oct 30 01:16:44.484488 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.484468 2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:16:44.488823 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.488805 2573 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:16:44.500498 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.500469 2573 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 01:16:44.504528 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.504515 2573 manager.go:341] "Starting Device Plugin manager" Oct 30 01:16:44.504595 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:44.504549 2573 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 01:16:44.504595 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.504561 2573 server.go:87] "Starting device plugin registration server" Oct 30 01:16:44.504821 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.504809 2573 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 01:16:44.504869 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.504823 2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 01:16:44.504948 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.504913 2573 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 01:16:44.505141 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.505126 2573 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 01:16:44.505141 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.505141 2573 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 01:16:44.505721 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:44.505699 2573 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 01:16:44.505795 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:44.505750 2573 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-65.ec2.internal\" not found" Oct 30 01:16:44.508578 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.508561 2573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:16:44.564612 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.564579 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 01:16:44.566473 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.565761 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 01:16:44.566538 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.566516 2573 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 01:16:44.566610 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.566552 2573 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 01:16:44.566610 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:44.566593 2573 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 01:16:44.579838 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.579816 2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:16:44.605196 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.605181 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 01:16:44.606111 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.606089 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-65.ec2.internal" event="NodeHasSufficientMemory" Oct 30 01:16:44.606210 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.606125 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-65.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 01:16:44.606210 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.606140 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-65.ec2.internal" event="NodeHasSufficientPID" Oct 30 01:16:44.606210 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.606166 2573 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-65.ec2.internal" Oct 30 01:16:44.624460 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.624396 2573 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-65.ec2.internal" Oct 30 01:16:44.624460 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:44.624420 2573 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-65.ec2.internal\": node \"ip-10-0-135-65.ec2.internal\" not found" Oct 30 01:16:44.666678 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.666648 2573 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-65.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-65.ec2.internal"] Oct 30 01:16:44.668988 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.668973 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-65.ec2.internal" Oct 30 01:16:44.670757 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.670743 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-65.ec2.internal" Oct 30 01:16:44.702592 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.702576 2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-65.ec2.internal" Oct 30 01:16:44.705138 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.705118 2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-65.ec2.internal" Oct 30 01:16:44.723121 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.723103 2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 01:16:44.726571 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.726558 2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 01:16:44.843449 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.843415 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a707a897ca34d8ee2c04b93cf900a7fe-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-65.ec2.internal\" (UID: \"a707a897ca34d8ee2c04b93cf900a7fe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-65.ec2.internal" Oct 30 01:16:44.843449 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.843452 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a707a897ca34d8ee2c04b93cf900a7fe-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-65.ec2.internal\" (UID: \"a707a897ca34d8ee2c04b93cf900a7fe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-65.ec2.internal" Oct 30 01:16:44.843643 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.843473 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7af5b4d87431a44e4b7050910f6ad38b-config\") pod \"kube-apiserver-proxy-ip-10-0-135-65.ec2.internal\" (UID: \"7af5b4d87431a44e4b7050910f6ad38b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-65.ec2.internal" Oct 30 01:16:44.944495 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.944413 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a707a897ca34d8ee2c04b93cf900a7fe-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-65.ec2.internal\" (UID: \"a707a897ca34d8ee2c04b93cf900a7fe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-65.ec2.internal" Oct 30 01:16:44.944495 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.944463 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a707a897ca34d8ee2c04b93cf900a7fe-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-65.ec2.internal\" (UID: \"a707a897ca34d8ee2c04b93cf900a7fe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-65.ec2.internal" Oct 30 01:16:44.944495 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.944463 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a707a897ca34d8ee2c04b93cf900a7fe-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-65.ec2.internal\" (UID: \"a707a897ca34d8ee2c04b93cf900a7fe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-65.ec2.internal" Oct 30 01:16:44.944682 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.944498 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7af5b4d87431a44e4b7050910f6ad38b-config\") pod \"kube-apiserver-proxy-ip-10-0-135-65.ec2.internal\" (UID: \"7af5b4d87431a44e4b7050910f6ad38b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-65.ec2.internal" Oct 30 01:16:44.944682 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.944542 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a707a897ca34d8ee2c04b93cf900a7fe-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-65.ec2.internal\" (UID: \"a707a897ca34d8ee2c04b93cf900a7fe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-65.ec2.internal" Oct 30 01:16:44.944682 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:44.944570 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7af5b4d87431a44e4b7050910f6ad38b-config\") pod \"kube-apiserver-proxy-ip-10-0-135-65.ec2.internal\" (UID: \"7af5b4d87431a44e4b7050910f6ad38b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-65.ec2.internal" Oct 30 01:16:45.026424 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:45.026398 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-65.ec2.internal" Oct 30 01:16:45.029469 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:45.029452 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-65.ec2.internal" Oct 30 01:16:46.432724 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.432561 2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 01:16:46.433164 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.432840 2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 01:16:46.433164 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.432851 2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 01:16:46.433164 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.432851 2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 01:16:46.472817 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.472791 2573 apiserver.go:52] "Watching apiserver" Oct 30 01:16:46.495217 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.495192 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 01:16:46.516662 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.516603 2573 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 01:16:46.516902 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.516882 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-pgk7v","openshift-ovn-kubernetes/ovnkube-node-2vsns","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs","openshift-dns/node-resolver-pf9jv","openshift-image-registry/node-ca-lm86p","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-65.ec2.internal","openshift-multus/multus-additional-cni-plugins-xfvmk","kube-system/global-pull-secret-syncer-dxz45","kube-system/kube-apiserver-proxy-ip-10-0-135-65.ec2.internal","openshift-multus/network-metrics-daemon-xj8mh","openshift-network-diagnostics/network-check-target-bbhx5","openshift-network-operator/iptables-alerter-mdwqw","kube-system/konnectivity-agent-9f9b4","openshift-cluster-node-tuning-operator/tuned-pkkfl"] Oct 30 01:16:46.519343 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.519325 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.523624 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.523605 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.527785 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.526780 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 01:11:45 +0000 UTC" deadline="2028-08-02 04:10:21.80313024 +0000 UTC" Oct 30 01:16:46.527785 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.526813 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15410h53m35.27632121s" Oct 30 01:16:46.527785 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.527253 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs" Oct 30 01:16:46.530250 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.530231 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pf9jv" Oct 30 01:16:46.532485 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.532466 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lm86p" Oct 30 01:16:46.534580 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.534560 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 01:16:46.534765 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.534621 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 01:16:46.534765 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.534705 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 01:16:46.534765 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.534730 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 01:16:46.534953 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.534783 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xfvmk" Oct 30 01:16:46.534953 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.534810 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fsgft\"" Oct 30 01:16:46.537145 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.537128 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxz45" Oct 30 01:16:46.537238 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:46.537196 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxz45" podUID="f10aab9c-d4e6-45b5-a703-146114a31329" Oct 30 01:16:46.538517 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.538341 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 01:16:46.538818 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.538803 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7dsj7\"" Oct 30 01:16:46.539460 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.539447 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xj8mh" Oct 30 01:16:46.539521 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:46.539504 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xj8mh" podUID="94864d90-1dd8-472e-a414-29c6b6a4f8f6" Oct 30 01:16:46.541193 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.541176 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 01:16:46.541640 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.541624 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 01:16:46.541966 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.541872 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbhx5" Oct 30 01:16:46.541966 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:46.541931 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbhx5" podUID="f4602ac2-652b-4026-9a0d-d72162bb3258" Oct 30 01:16:46.543260 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.543239 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 01:16:46.543373 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.543319 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 01:16:46.543525 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.543514 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 01:16:46.543652 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.543640 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 01:16:46.543833 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.543819 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-p4hjp\"" Oct 30 01:16:46.543994 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.543982 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 01:16:46.544111 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.544102 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mdwqw" Oct 30 01:16:46.544437 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.544417 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 01:16:46.545697 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.545683 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 01:16:46.546292 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.546270 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9f9b4" Oct 30 01:16:46.547270 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.547250 2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 01:16:46.547347 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.547318 2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5db2r" Oct 30 01:16:46.548504 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.548490 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:16:46.553836 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.553816 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5db2r" Oct 30 01:16:46.597255 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.597234 2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 01:16:46.601179 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.601155 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdvvv\" (UniqueName: \"kubernetes.io/projected/ecb9d3db-582f-41a0-945d-7bd7e4892ff8-kube-api-access-kdvvv\") pod \"node-ca-lm86p\" (UID: \"ecb9d3db-582f-41a0-945d-7bd7e4892ff8\") " pod="openshift-image-registry/node-ca-lm86p" Oct 30 01:16:46.601292 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.601189 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2ec1edc-b911-4537-ac00-f708ef6c9062-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xfvmk\" (UID: \"a2ec1edc-b911-4537-ac00-f708ef6c9062\") " pod="openshift-multus/multus-additional-cni-plugins-xfvmk" Oct 30 01:16:46.601292 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.601218 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-etc-kubernetes\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.601292 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.601244 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-run-openvswitch\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.601292 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.601266 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-log-socket\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.601292 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.601287 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-host-run-ovn-kubernetes\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.601512 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.601319 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w5t7\" (UniqueName: \"kubernetes.io/projected/d332bbb4-c699-44e8-96ba-516d8ca03e95-kube-api-access-8w5t7\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.601512 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.601343 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bda81d83-62da-4f40-b668-29e2b0e51f35-multus-daemon-config\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.601512 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.601387 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5618fce7-e2ca-4423-887a-eea73fdd9d01-iptables-alerter-script\") pod \"iptables-alerter-mdwqw\" (UID: \"5618fce7-e2ca-4423-887a-eea73fdd9d01\") " pod="openshift-network-operator/iptables-alerter-mdwqw" Oct 30 01:16:46.601512 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.601416 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e30ba693-6284-485e-95fd-ad639627aa8d-agent-certs\") pod \"konnectivity-agent-9f9b4\" (UID: \"e30ba693-6284-485e-95fd-ad639627aa8d\") " pod="kube-system/konnectivity-agent-9f9b4" Oct 30 01:16:46.601512 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.601436 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e09d6f2e-7cd5-47c3-b18b-482726067e90-tmp\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:16:46.601512 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.601463 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-host-kubelet\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.601512 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.601495 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-etc-openvswitch\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.601813 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.601538 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89k2s\" (UniqueName: \"kubernetes.io/projected/f4602ac2-652b-4026-9a0d-d72162bb3258-kube-api-access-89k2s\") pod \"network-check-target-bbhx5\" (UID: \"f4602ac2-652b-4026-9a0d-d72162bb3258\") " pod="openshift-network-diagnostics/network-check-target-bbhx5" Oct 30 01:16:46.601813 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.601559 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-multus-socket-dir-parent\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.601813 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.601576 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f10aab9c-d4e6-45b5-a703-146114a31329-kubelet-config\") pod \"global-pull-secret-syncer-dxz45\" (UID: \"f10aab9c-d4e6-45b5-a703-146114a31329\") " pod="kube-system/global-pull-secret-syncer-dxz45" Oct 30 01:16:46.601813 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.601589 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-node-log\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.601813 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.601606 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2ec1edc-b911-4537-ac00-f708ef6c9062-cnibin\") pod \"multus-additional-cni-plugins-xfvmk\" (UID: \"a2ec1edc-b911-4537-ac00-f708ef6c9062\") " pod="openshift-multus/multus-additional-cni-plugins-xfvmk" Oct 30 01:16:46.601813 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.601653 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-etc-modprobe-d\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:16:46.601813 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.601683 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-multus-cni-dir\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.601813 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.601709 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c819c74a-0f8d-4a5e-9e1b-830196f6b117-tmp-dir\") pod \"node-resolver-pf9jv\" (UID: \"c819c74a-0f8d-4a5e-9e1b-830196f6b117\") " pod="openshift-dns/node-resolver-pf9jv" Oct 30 01:16:46.601813 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.601749 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f10aab9c-d4e6-45b5-a703-146114a31329-original-pull-secret\") pod \"global-pull-secret-syncer-dxz45\" (UID: \"f10aab9c-d4e6-45b5-a703-146114a31329\") " pod="kube-system/global-pull-secret-syncer-dxz45" Oct 30 01:16:46.601813 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.601784 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d332bbb4-c699-44e8-96ba-516d8ca03e95-ovnkube-script-lib\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.602171 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.601829 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-cnibin\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.602171 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.601872 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rmpm\" (UniqueName: \"kubernetes.io/projected/94864d90-1dd8-472e-a414-29c6b6a4f8f6-kube-api-access-5rmpm\") pod \"network-metrics-daemon-xj8mh\" (UID: \"94864d90-1dd8-472e-a414-29c6b6a4f8f6\") " pod="openshift-multus/network-metrics-daemon-xj8mh" Oct 30 01:16:46.602171 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.601896 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a4365b4f-a9d0-41eb-9d89-26591d3e3ed5-device-dir\") pod \"aws-ebs-csi-driver-node-2wbfs\" (UID: \"a4365b4f-a9d0-41eb-9d89-26591d3e3ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs" Oct 30 01:16:46.602171 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.601930 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a2ec1edc-b911-4537-ac00-f708ef6c9062-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xfvmk\" (UID: \"a2ec1edc-b911-4537-ac00-f708ef6c9062\") " pod="openshift-multus/multus-additional-cni-plugins-xfvmk" Oct 30 01:16:46.602171 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.601976 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-var-lib-kubelet\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:16:46.602171 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.602012 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e30ba693-6284-485e-95fd-ad639627aa8d-konnectivity-ca\") pod \"konnectivity-agent-9f9b4\" (UID: \"e30ba693-6284-485e-95fd-ad639627aa8d\") " pod="kube-system/konnectivity-agent-9f9b4" Oct 30 01:16:46.602171 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.602034 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-run-systemd\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.602171 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.602050 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d332bbb4-c699-44e8-96ba-516d8ca03e95-ovn-node-metrics-cert\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.602171 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.602066 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-etc-systemd\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:16:46.602171 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.602085 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-run-ovn\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.602171 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.602100 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmptt\" (UniqueName: \"kubernetes.io/projected/e09d6f2e-7cd5-47c3-b18b-482726067e90-kube-api-access-nmptt\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:16:46.602897 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.602878 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-host-var-lib-kubelet\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.602977 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.602919 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a4365b4f-a9d0-41eb-9d89-26591d3e3ed5-socket-dir\") pod \"aws-ebs-csi-driver-node-2wbfs\" (UID: \"a4365b4f-a9d0-41eb-9d89-26591d3e3ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs" Oct 30 01:16:46.602977 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.602946 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ecb9d3db-582f-41a0-945d-7bd7e4892ff8-host\") pod \"node-ca-lm86p\" (UID: \"ecb9d3db-582f-41a0-945d-7bd7e4892ff8\") " pod="openshift-image-registry/node-ca-lm86p" Oct 30 01:16:46.603082 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.602998 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2ec1edc-b911-4537-ac00-f708ef6c9062-cni-binary-copy\") pod \"multus-additional-cni-plugins-xfvmk\" (UID: \"a2ec1edc-b911-4537-ac00-f708ef6c9062\") " pod="openshift-multus/multus-additional-cni-plugins-xfvmk" Oct 30 01:16:46.603082 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.603032 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qbvg\" (UniqueName: \"kubernetes.io/projected/a2ec1edc-b911-4537-ac00-f708ef6c9062-kube-api-access-8qbvg\") pod \"multus-additional-cni-plugins-xfvmk\" (UID: \"a2ec1edc-b911-4537-ac00-f708ef6c9062\") " pod="openshift-multus/multus-additional-cni-plugins-xfvmk" Oct 30 01:16:46.603082 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.603064 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d332bbb4-c699-44e8-96ba-516d8ca03e95-ovnkube-config\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.603226 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.603090 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-etc-kubernetes\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:16:46.603226 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.603138 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-etc-sysctl-conf\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:16:46.603324 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.603219 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-lib-modules\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:16:46.603324 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.603253 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-host-slash\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.603324 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.603277 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-host-cni-bin\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.603324 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.603302 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-host-cni-netd\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.603500 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.603327 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a4365b4f-a9d0-41eb-9d89-26591d3e3ed5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2wbfs\" (UID: \"a4365b4f-a9d0-41eb-9d89-26591d3e3ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs" Oct 30 01:16:46.603500 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.603364 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ecb9d3db-582f-41a0-945d-7bd7e4892ff8-serviceca\") pod \"node-ca-lm86p\" (UID: \"ecb9d3db-582f-41a0-945d-7bd7e4892ff8\") " pod="openshift-image-registry/node-ca-lm86p" Oct 30 01:16:46.603500 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.603383 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a2ec1edc-b911-4537-ac00-f708ef6c9062-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xfvmk\" (UID: \"a2ec1edc-b911-4537-ac00-f708ef6c9062\") " pod="openshift-multus/multus-additional-cni-plugins-xfvmk" Oct 30 01:16:46.603500 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.603422 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-etc-sysctl-d\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:16:46.603500 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.603463 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-host-run-netns\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.603695 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.603507 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a4365b4f-a9d0-41eb-9d89-26591d3e3ed5-registration-dir\") pod \"aws-ebs-csi-driver-node-2wbfs\" (UID: \"a4365b4f-a9d0-41eb-9d89-26591d3e3ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs" Oct 30 01:16:46.603695 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.603545 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-sys\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:16:46.603695 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.603580 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-host-var-lib-cni-multus\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.603695 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.603606 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5wrm\" (UniqueName: \"kubernetes.io/projected/bda81d83-62da-4f40-b668-29e2b0e51f35-kube-api-access-p5wrm\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.603695 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.603653 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-systemd-units\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.603695 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.603673 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.603695 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.603689 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2ec1edc-b911-4537-ac00-f708ef6c9062-system-cni-dir\") pod \"multus-additional-cni-plugins-xfvmk\" (UID: \"a2ec1edc-b911-4537-ac00-f708ef6c9062\") " pod="openshift-multus/multus-additional-cni-plugins-xfvmk" Oct 30 01:16:46.603913 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.603704 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bda81d83-62da-4f40-b668-29e2b0e51f35-cni-binary-copy\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.603913 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.603728 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-hostroot\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.603913 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.603757 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dhns\" (UniqueName: \"kubernetes.io/projected/c819c74a-0f8d-4a5e-9e1b-830196f6b117-kube-api-access-4dhns\") pod \"node-resolver-pf9jv\" (UID: \"c819c74a-0f8d-4a5e-9e1b-830196f6b117\") " pod="openshift-dns/node-resolver-pf9jv" Oct 30 01:16:46.603913 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.603771 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f10aab9c-d4e6-45b5-a703-146114a31329-dbus\") pod \"global-pull-secret-syncer-dxz45\" (UID: \"f10aab9c-d4e6-45b5-a703-146114a31329\") " pod="kube-system/global-pull-secret-syncer-dxz45" Oct 30 01:16:46.603913 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.603797 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7pbf\" (UniqueName: \"kubernetes.io/projected/5618fce7-e2ca-4423-887a-eea73fdd9d01-kube-api-access-n7pbf\") pod \"iptables-alerter-mdwqw\" (UID: \"5618fce7-e2ca-4423-887a-eea73fdd9d01\") " pod="openshift-network-operator/iptables-alerter-mdwqw" Oct 30 01:16:46.603913 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.603811 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a4365b4f-a9d0-41eb-9d89-26591d3e3ed5-etc-selinux\") pod \"aws-ebs-csi-driver-node-2wbfs\" (UID: \"a4365b4f-a9d0-41eb-9d89-26591d3e3ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs" Oct 30 01:16:46.603913 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.603825 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d332bbb4-c699-44e8-96ba-516d8ca03e95-env-overrides\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.603913 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.603849 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f67fc\" (UniqueName: \"kubernetes.io/projected/a4365b4f-a9d0-41eb-9d89-26591d3e3ed5-kube-api-access-f67fc\") pod \"aws-ebs-csi-driver-node-2wbfs\" (UID: \"a4365b4f-a9d0-41eb-9d89-26591d3e3ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs" Oct 30 01:16:46.603913 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.603873 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-host-run-multus-certs\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.603913 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.603897 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-var-lib-openvswitch\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.603913 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.603912 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a4365b4f-a9d0-41eb-9d89-26591d3e3ed5-sys-fs\") pod \"aws-ebs-csi-driver-node-2wbfs\" (UID: \"a4365b4f-a9d0-41eb-9d89-26591d3e3ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs" Oct 30 01:16:46.604237 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.603925 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-run\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:16:46.604237 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.603939 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-host\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:16:46.604237 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.603958 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-system-cni-dir\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.604237 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.603973 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-host-run-k8s-cni-cncf-io\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.604237 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.603986 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-host-run-netns\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.604237 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.604004 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-etc-sysconfig\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:16:46.604237 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.604043 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e09d6f2e-7cd5-47c3-b18b-482726067e90-etc-tuned\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:16:46.604237 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.604072 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-host-var-lib-cni-bin\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.604237 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.604102 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-multus-conf-dir\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.604237 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.604124 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c819c74a-0f8d-4a5e-9e1b-830196f6b117-hosts-file\") pod \"node-resolver-pf9jv\" (UID: \"c819c74a-0f8d-4a5e-9e1b-830196f6b117\") " pod="openshift-dns/node-resolver-pf9jv" Oct 30 01:16:46.604237 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.604141 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94864d90-1dd8-472e-a414-29c6b6a4f8f6-metrics-certs\") pod \"network-metrics-daemon-xj8mh\" (UID: \"94864d90-1dd8-472e-a414-29c6b6a4f8f6\") " pod="openshift-multus/network-metrics-daemon-xj8mh" Oct 30 01:16:46.604237 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.604159 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5618fce7-e2ca-4423-887a-eea73fdd9d01-host-slash\") pod \"iptables-alerter-mdwqw\" (UID: \"5618fce7-e2ca-4423-887a-eea73fdd9d01\") " pod="openshift-network-operator/iptables-alerter-mdwqw" Oct 30 01:16:46.638081 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.638059 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 01:16:46.638175 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.638068 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 01:16:46.638175 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.638115 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pcscp\"" Oct 30 01:16:46.638242 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.638191 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-p2b59\"" Oct 30 01:16:46.639665 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.639653 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 01:16:46.639830 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.639820 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 01:16:46.640267 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.640251 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hb7tb\"" Oct 30 01:16:46.640329 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.640265 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-58lhf\"" Oct 30 01:16:46.640329 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.640319 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 01:16:46.644847 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.644828 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 01:16:46.645340 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.645315 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 01:16:46.645434 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.645378 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j8bsj\"" Oct 30 01:16:46.645434 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.645412 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 01:16:46.645558 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.645447 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 01:16:46.645558 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.645476 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 01:16:46.645633 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.645609 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 01:16:46.645665 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.645654 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 01:16:46.645739 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.645725 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-srg4h\"" Oct 30 01:16:46.645781 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.645770 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 01:16:46.646784 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:46.646755 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda707a897ca34d8ee2c04b93cf900a7fe.slice/crio-e357503c9c89fb4c2823d4997882a3185c18d2abb7aa618b6b31f657e077742d WatchSource:0}: Error finding container e357503c9c89fb4c2823d4997882a3185c18d2abb7aa618b6b31f657e077742d: Status 404 returned error can't find the container with id e357503c9c89fb4c2823d4997882a3185c18d2abb7aa618b6b31f657e077742d Oct 30 01:16:46.647208 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:46.647192 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7af5b4d87431a44e4b7050910f6ad38b.slice/crio-415e62159ff51389db9ac1f6f638080802e928cef05b934bed5ae6c9b0e64520 WatchSource:0}: Error finding container 415e62159ff51389db9ac1f6f638080802e928cef05b934bed5ae6c9b0e64520: Status 404 returned error can't find the container with id 415e62159ff51389db9ac1f6f638080802e928cef05b934bed5ae6c9b0e64520 Oct 30 01:16:46.651801 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.651780 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:16:46.705052 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.705025 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-run-openvswitch\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.705052 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.705055 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-log-socket\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.705239 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.705071 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-host-run-ovn-kubernetes\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.705239 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.705085 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w5t7\" (UniqueName: \"kubernetes.io/projected/d332bbb4-c699-44e8-96ba-516d8ca03e95-kube-api-access-8w5t7\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.705239 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.705114 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bda81d83-62da-4f40-b668-29e2b0e51f35-multus-daemon-config\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.705239 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.705149 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-log-socket\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.705239 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.705162 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-host-run-ovn-kubernetes\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.705239 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.705175 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-run-openvswitch\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.705239 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.705201 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5618fce7-e2ca-4423-887a-eea73fdd9d01-iptables-alerter-script\") pod \"iptables-alerter-mdwqw\" (UID: \"5618fce7-e2ca-4423-887a-eea73fdd9d01\") " pod="openshift-network-operator/iptables-alerter-mdwqw" Oct 30 01:16:46.705239 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.705233 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e30ba693-6284-485e-95fd-ad639627aa8d-agent-certs\") pod \"konnectivity-agent-9f9b4\" (UID: \"e30ba693-6284-485e-95fd-ad639627aa8d\") " pod="kube-system/konnectivity-agent-9f9b4" Oct 30 01:16:46.705585 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.705255 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e09d6f2e-7cd5-47c3-b18b-482726067e90-tmp\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:16:46.705585 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.705282 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-host-kubelet\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.705585 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.705307 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-etc-openvswitch\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.705585 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.705351 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89k2s\" (UniqueName: \"kubernetes.io/projected/f4602ac2-652b-4026-9a0d-d72162bb3258-kube-api-access-89k2s\") pod \"network-check-target-bbhx5\" (UID: \"f4602ac2-652b-4026-9a0d-d72162bb3258\") " pod="openshift-network-diagnostics/network-check-target-bbhx5" Oct 30 01:16:46.705585 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.705375 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-multus-socket-dir-parent\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.705585 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.705397 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f10aab9c-d4e6-45b5-a703-146114a31329-kubelet-config\") pod \"global-pull-secret-syncer-dxz45\" (UID: \"f10aab9c-d4e6-45b5-a703-146114a31329\") " pod="kube-system/global-pull-secret-syncer-dxz45" Oct 30 01:16:46.705585 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.705397 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-host-kubelet\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.705585 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.705470 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-node-log\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.705585 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.705418 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-node-log\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.705585 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.705484 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f10aab9c-d4e6-45b5-a703-146114a31329-kubelet-config\") pod \"global-pull-secret-syncer-dxz45\" (UID: \"f10aab9c-d4e6-45b5-a703-146114a31329\") " pod="kube-system/global-pull-secret-syncer-dxz45" Oct 30 01:16:46.705585 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.705407 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-etc-openvswitch\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.705585 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.705515 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2ec1edc-b911-4537-ac00-f708ef6c9062-cnibin\") pod \"multus-additional-cni-plugins-xfvmk\" (UID: \"a2ec1edc-b911-4537-ac00-f708ef6c9062\") " pod="openshift-multus/multus-additional-cni-plugins-xfvmk" Oct 30 01:16:46.705585 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.705515 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-multus-socket-dir-parent\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.705585 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.705567 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2ec1edc-b911-4537-ac00-f708ef6c9062-cnibin\") pod \"multus-additional-cni-plugins-xfvmk\" (UID: \"a2ec1edc-b911-4537-ac00-f708ef6c9062\") " pod="openshift-multus/multus-additional-cni-plugins-xfvmk" Oct 30 01:16:46.705585 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.705569 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-etc-modprobe-d\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:16:46.706289 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.705603 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-multus-cni-dir\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.706289 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.705628 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c819c74a-0f8d-4a5e-9e1b-830196f6b117-tmp-dir\") pod \"node-resolver-pf9jv\" (UID: \"c819c74a-0f8d-4a5e-9e1b-830196f6b117\") " pod="openshift-dns/node-resolver-pf9jv" Oct 30 01:16:46.706289 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.705648 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-etc-modprobe-d\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:16:46.706289 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.705652 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f10aab9c-d4e6-45b5-a703-146114a31329-original-pull-secret\") pod \"global-pull-secret-syncer-dxz45\" (UID: \"f10aab9c-d4e6-45b5-a703-146114a31329\") " pod="kube-system/global-pull-secret-syncer-dxz45" Oct 30 01:16:46.706289 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.705679 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d332bbb4-c699-44e8-96ba-516d8ca03e95-ovnkube-script-lib\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.706289 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.705702 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-cnibin\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.706289 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.705725 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rmpm\" (UniqueName: \"kubernetes.io/projected/94864d90-1dd8-472e-a414-29c6b6a4f8f6-kube-api-access-5rmpm\") pod \"network-metrics-daemon-xj8mh\" (UID: \"94864d90-1dd8-472e-a414-29c6b6a4f8f6\") " pod="openshift-multus/network-metrics-daemon-xj8mh" Oct 30 01:16:46.706289 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.705752 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-multus-cni-dir\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.706289 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.705752 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a4365b4f-a9d0-41eb-9d89-26591d3e3ed5-device-dir\") pod \"aws-ebs-csi-driver-node-2wbfs\" (UID: \"a4365b4f-a9d0-41eb-9d89-26591d3e3ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs" Oct 30 01:16:46.706289 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.705775 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-cnibin\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.706289 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.705784 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a2ec1edc-b911-4537-ac00-f708ef6c9062-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xfvmk\" (UID: \"a2ec1edc-b911-4537-ac00-f708ef6c9062\") " pod="openshift-multus/multus-additional-cni-plugins-xfvmk" Oct 30 01:16:46.706289 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.705804 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-var-lib-kubelet\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:16:46.706289 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.705821 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e30ba693-6284-485e-95fd-ad639627aa8d-konnectivity-ca\") pod \"konnectivity-agent-9f9b4\" (UID: \"e30ba693-6284-485e-95fd-ad639627aa8d\") " pod="kube-system/konnectivity-agent-9f9b4" Oct 30 01:16:46.706289 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.705835 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-run-systemd\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.706289 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:46.705852 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 01:16:46.706289 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.705865 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d332bbb4-c699-44e8-96ba-516d8ca03e95-ovn-node-metrics-cert\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.706289 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.705880 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-etc-systemd\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:16:46.706289 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.705899 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-run-ovn\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.707102 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.705922 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmptt\" (UniqueName: \"kubernetes.io/projected/e09d6f2e-7cd5-47c3-b18b-482726067e90-kube-api-access-nmptt\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:16:46.707102 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.705943 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-host-var-lib-kubelet\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.707102 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.705965 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a4365b4f-a9d0-41eb-9d89-26591d3e3ed5-socket-dir\") pod \"aws-ebs-csi-driver-node-2wbfs\" (UID: \"a4365b4f-a9d0-41eb-9d89-26591d3e3ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs" Oct 30 01:16:46.707102 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.705987 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ecb9d3db-582f-41a0-945d-7bd7e4892ff8-host\") pod \"node-ca-lm86p\" (UID: \"ecb9d3db-582f-41a0-945d-7bd7e4892ff8\") " pod="openshift-image-registry/node-ca-lm86p" Oct 30 01:16:46.707102 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.706008 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2ec1edc-b911-4537-ac00-f708ef6c9062-cni-binary-copy\") pod \"multus-additional-cni-plugins-xfvmk\" (UID: \"a2ec1edc-b911-4537-ac00-f708ef6c9062\") " pod="openshift-multus/multus-additional-cni-plugins-xfvmk" Oct 30 01:16:46.707102 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.706004 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-etc-systemd\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:16:46.707102 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.706035 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qbvg\" (UniqueName: \"kubernetes.io/projected/a2ec1edc-b911-4537-ac00-f708ef6c9062-kube-api-access-8qbvg\") pod \"multus-additional-cni-plugins-xfvmk\" (UID: \"a2ec1edc-b911-4537-ac00-f708ef6c9062\") " pod="openshift-multus/multus-additional-cni-plugins-xfvmk" Oct 30 01:16:46.707102 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.706072 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d332bbb4-c699-44e8-96ba-516d8ca03e95-ovnkube-config\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.707102 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.706101 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-etc-kubernetes\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:16:46.707102 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.706127 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-etc-sysctl-conf\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:16:46.707102 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.706151 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-lib-modules\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:16:46.707102 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.706139 2573 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 01:16:46.707102 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.706180 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-host-slash\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.707102 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:46.706235 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f10aab9c-d4e6-45b5-a703-146114a31329-original-pull-secret podName:f10aab9c-d4e6-45b5-a703-146114a31329 nodeName:}" failed. No retries permitted until 2026-10-30 01:16:47.20619785 +0000 UTC m=+2.102175822 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f10aab9c-d4e6-45b5-a703-146114a31329-original-pull-secret") pod "global-pull-secret-syncer-dxz45" (UID: "f10aab9c-d4e6-45b5-a703-146114a31329") : object "kube-system"/"original-pull-secret" not registered Oct 30 01:16:46.707102 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.706272 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-host-cni-bin\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.707102 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.706282 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-run-systemd\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.707102 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.706324 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a4365b4f-a9d0-41eb-9d89-26591d3e3ed5-socket-dir\") pod \"aws-ebs-csi-driver-node-2wbfs\" (UID: \"a4365b4f-a9d0-41eb-9d89-26591d3e3ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs" Oct 30 01:16:46.707866 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.706352 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bda81d83-62da-4f40-b668-29e2b0e51f35-multus-daemon-config\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.707866 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.706368 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ecb9d3db-582f-41a0-945d-7bd7e4892ff8-host\") pod \"node-ca-lm86p\" (UID: \"ecb9d3db-582f-41a0-945d-7bd7e4892ff8\") " pod="openshift-image-registry/node-ca-lm86p" Oct 30 01:16:46.707866 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.706371 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-host-cni-netd\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.707866 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.706403 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a4365b4f-a9d0-41eb-9d89-26591d3e3ed5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2wbfs\" (UID: \"a4365b4f-a9d0-41eb-9d89-26591d3e3ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs" Oct 30 01:16:46.707866 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.706443 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ecb9d3db-582f-41a0-945d-7bd7e4892ff8-serviceca\") pod \"node-ca-lm86p\" (UID: \"ecb9d3db-582f-41a0-945d-7bd7e4892ff8\") " pod="openshift-image-registry/node-ca-lm86p" Oct 30 01:16:46.707866 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.706469 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a2ec1edc-b911-4537-ac00-f708ef6c9062-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xfvmk\" (UID: \"a2ec1edc-b911-4537-ac00-f708ef6c9062\") " pod="openshift-multus/multus-additional-cni-plugins-xfvmk" Oct 30 01:16:46.707866 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.706474 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e30ba693-6284-485e-95fd-ad639627aa8d-konnectivity-ca\") pod \"konnectivity-agent-9f9b4\" (UID: \"e30ba693-6284-485e-95fd-ad639627aa8d\") " pod="kube-system/konnectivity-agent-9f9b4" Oct 30 01:16:46.707866 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.706497 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-etc-sysctl-d\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:16:46.707866 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.706550 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-host-run-netns\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.707866 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.706573 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a4365b4f-a9d0-41eb-9d89-26591d3e3ed5-registration-dir\") pod \"aws-ebs-csi-driver-node-2wbfs\" (UID: \"a4365b4f-a9d0-41eb-9d89-26591d3e3ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs" Oct 30 01:16:46.707866 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.706601 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a2ec1edc-b911-4537-ac00-f708ef6c9062-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xfvmk\" (UID: \"a2ec1edc-b911-4537-ac00-f708ef6c9062\") " pod="openshift-multus/multus-additional-cni-plugins-xfvmk" Oct 30 01:16:46.707866 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.706623 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-sys\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:16:46.707866 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.706648 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-host-var-lib-cni-multus\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.707866 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.706684 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5wrm\" (UniqueName: \"kubernetes.io/projected/bda81d83-62da-4f40-b668-29e2b0e51f35-kube-api-access-p5wrm\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.707866 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.706725 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-systemd-units\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.707866 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.706744 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-etc-sysctl-conf\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:16:46.707866 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.706762 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.708666 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.706786 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2ec1edc-b911-4537-ac00-f708ef6c9062-system-cni-dir\") pod \"multus-additional-cni-plugins-xfvmk\" (UID: \"a2ec1edc-b911-4537-ac00-f708ef6c9062\") " pod="openshift-multus/multus-additional-cni-plugins-xfvmk" Oct 30 01:16:46.708666 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.706806 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bda81d83-62da-4f40-b668-29e2b0e51f35-cni-binary-copy\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.708666 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.706839 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-hostroot\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.708666 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.706861 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dhns\" (UniqueName: \"kubernetes.io/projected/c819c74a-0f8d-4a5e-9e1b-830196f6b117-kube-api-access-4dhns\") pod \"node-resolver-pf9jv\" (UID: \"c819c74a-0f8d-4a5e-9e1b-830196f6b117\") " pod="openshift-dns/node-resolver-pf9jv" Oct 30 01:16:46.708666 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.706871 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2ec1edc-b911-4537-ac00-f708ef6c9062-cni-binary-copy\") pod \"multus-additional-cni-plugins-xfvmk\" (UID: \"a2ec1edc-b911-4537-ac00-f708ef6c9062\") " pod="openshift-multus/multus-additional-cni-plugins-xfvmk" Oct 30 01:16:46.708666 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.706884 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f10aab9c-d4e6-45b5-a703-146114a31329-dbus\") pod \"global-pull-secret-syncer-dxz45\" (UID: \"f10aab9c-d4e6-45b5-a703-146114a31329\") " pod="kube-system/global-pull-secret-syncer-dxz45" Oct 30 01:16:46.708666 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.706916 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7pbf\" (UniqueName: \"kubernetes.io/projected/5618fce7-e2ca-4423-887a-eea73fdd9d01-kube-api-access-n7pbf\") pod \"iptables-alerter-mdwqw\" (UID: \"5618fce7-e2ca-4423-887a-eea73fdd9d01\") " pod="openshift-network-operator/iptables-alerter-mdwqw" Oct 30 01:16:46.708666 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.706941 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a4365b4f-a9d0-41eb-9d89-26591d3e3ed5-etc-selinux\") pod \"aws-ebs-csi-driver-node-2wbfs\" (UID: \"a4365b4f-a9d0-41eb-9d89-26591d3e3ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs" Oct 30 01:16:46.708666 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.706951 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f10aab9c-d4e6-45b5-a703-146114a31329-dbus\") pod \"global-pull-secret-syncer-dxz45\" (UID: \"f10aab9c-d4e6-45b5-a703-146114a31329\") " pod="kube-system/global-pull-secret-syncer-dxz45" Oct 30 01:16:46.708666 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.706973 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d332bbb4-c699-44e8-96ba-516d8ca03e95-ovnkube-script-lib\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.708666 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.706987 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d332bbb4-c699-44e8-96ba-516d8ca03e95-env-overrides\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.708666 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.707015 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f67fc\" (UniqueName: \"kubernetes.io/projected/a4365b4f-a9d0-41eb-9d89-26591d3e3ed5-kube-api-access-f67fc\") pod \"aws-ebs-csi-driver-node-2wbfs\" (UID: \"a4365b4f-a9d0-41eb-9d89-26591d3e3ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs" Oct 30 01:16:46.708666 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.707019 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-host-cni-bin\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.708666 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.707039 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-host-run-multus-certs\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.708666 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.707067 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-host-cni-netd\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.708666 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.707069 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-var-lib-openvswitch\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.708666 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.707096 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a4365b4f-a9d0-41eb-9d89-26591d3e3ed5-sys-fs\") pod \"aws-ebs-csi-driver-node-2wbfs\" (UID: \"a4365b4f-a9d0-41eb-9d89-26591d3e3ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs" Oct 30 01:16:46.709489 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.707108 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-var-lib-openvswitch\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.709489 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.707117 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-run\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:16:46.709489 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.707159 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-host\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:16:46.709489 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.707153 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d332bbb4-c699-44e8-96ba-516d8ca03e95-ovnkube-config\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.709489 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.707180 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-system-cni-dir\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.709489 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.707206 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-host-run-k8s-cni-cncf-io\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.709489 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.707241 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-host-run-netns\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.709489 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.707262 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-etc-sysconfig\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:16:46.709489 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.707274 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-etc-kubernetes\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:16:46.709489 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.707281 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e09d6f2e-7cd5-47c3-b18b-482726067e90-etc-tuned\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:16:46.709489 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.707317 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-host-var-lib-cni-bin\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.709489 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.707327 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c819c74a-0f8d-4a5e-9e1b-830196f6b117-tmp-dir\") pod \"node-resolver-pf9jv\" (UID: \"c819c74a-0f8d-4a5e-9e1b-830196f6b117\") " pod="openshift-dns/node-resolver-pf9jv" Oct 30 01:16:46.709489 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.707338 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-multus-conf-dir\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.709489 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.707346 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-var-lib-kubelet\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:16:46.709489 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.707359 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c819c74a-0f8d-4a5e-9e1b-830196f6b117-hosts-file\") pod \"node-resolver-pf9jv\" (UID: \"c819c74a-0f8d-4a5e-9e1b-830196f6b117\") " pod="openshift-dns/node-resolver-pf9jv" Oct 30 01:16:46.709489 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.707393 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94864d90-1dd8-472e-a414-29c6b6a4f8f6-metrics-certs\") pod \"network-metrics-daemon-xj8mh\" (UID: \"94864d90-1dd8-472e-a414-29c6b6a4f8f6\") " pod="openshift-multus/network-metrics-daemon-xj8mh" Oct 30 01:16:46.709489 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.707423 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5618fce7-e2ca-4423-887a-eea73fdd9d01-host-slash\") pod \"iptables-alerter-mdwqw\" (UID: \"5618fce7-e2ca-4423-887a-eea73fdd9d01\") " pod="openshift-network-operator/iptables-alerter-mdwqw" Oct 30 01:16:46.709489 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.707433 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c819c74a-0f8d-4a5e-9e1b-830196f6b117-hosts-file\") pod \"node-resolver-pf9jv\" (UID: \"c819c74a-0f8d-4a5e-9e1b-830196f6b117\") " pod="openshift-dns/node-resolver-pf9jv" Oct 30 01:16:46.710288 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.707450 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdvvv\" (UniqueName: \"kubernetes.io/projected/ecb9d3db-582f-41a0-945d-7bd7e4892ff8-kube-api-access-kdvvv\") pod \"node-ca-lm86p\" (UID: \"ecb9d3db-582f-41a0-945d-7bd7e4892ff8\") " pod="openshift-image-registry/node-ca-lm86p" Oct 30 01:16:46.710288 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.707486 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-hostroot\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.710288 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.707509 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2ec1edc-b911-4537-ac00-f708ef6c9062-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xfvmk\" (UID: \"a2ec1edc-b911-4537-ac00-f708ef6c9062\") " pod="openshift-multus/multus-additional-cni-plugins-xfvmk" Oct 30 01:16:46.710288 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.707562 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-etc-kubernetes\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.710288 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.707639 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-etc-kubernetes\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.710288 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.707651 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d332bbb4-c699-44e8-96ba-516d8ca03e95-env-overrides\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.710288 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.706224 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-run-ovn\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.710288 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.707693 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5618fce7-e2ca-4423-887a-eea73fdd9d01-iptables-alerter-script\") pod \"iptables-alerter-mdwqw\" (UID: \"5618fce7-e2ca-4423-887a-eea73fdd9d01\") " pod="openshift-network-operator/iptables-alerter-mdwqw" Oct 30 01:16:46.710288 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.707697 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5618fce7-e2ca-4423-887a-eea73fdd9d01-host-slash\") pod \"iptables-alerter-mdwqw\" (UID: \"5618fce7-e2ca-4423-887a-eea73fdd9d01\") " pod="openshift-network-operator/iptables-alerter-mdwqw" Oct 30 01:16:46.710288 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.707715 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a4365b4f-a9d0-41eb-9d89-26591d3e3ed5-sys-fs\") pod \"aws-ebs-csi-driver-node-2wbfs\" (UID: \"a4365b4f-a9d0-41eb-9d89-26591d3e3ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs" Oct 30 01:16:46.710288 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.707775 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a4365b4f-a9d0-41eb-9d89-26591d3e3ed5-registration-dir\") pod \"aws-ebs-csi-driver-node-2wbfs\" (UID: \"a4365b4f-a9d0-41eb-9d89-26591d3e3ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs" Oct 30 01:16:46.710288 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.707801 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bda81d83-62da-4f40-b668-29e2b0e51f35-cni-binary-copy\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.710288 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.707802 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-lib-modules\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:16:46.710288 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.707821 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a4365b4f-a9d0-41eb-9d89-26591d3e3ed5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2wbfs\" (UID: \"a4365b4f-a9d0-41eb-9d89-26591d3e3ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs" Oct 30 01:16:46.710288 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.707888 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-host-slash\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.710288 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.706177 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-host-var-lib-kubelet\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.710288 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.707949 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.710288 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.707976 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-run\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:16:46.710832 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.707994 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-systemd-units\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.710832 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.708034 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-host\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:16:46.710832 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.708036 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-host-var-lib-cni-multus\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.710832 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.708083 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-host-run-k8s-cni-cncf-io\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.710832 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.708099 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-system-cni-dir\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.710832 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.708041 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-sys\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:16:46.710832 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.708121 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-host-run-netns\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.710832 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.708130 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a4365b4f-a9d0-41eb-9d89-26591d3e3ed5-etc-selinux\") pod \"aws-ebs-csi-driver-node-2wbfs\" (UID: \"a4365b4f-a9d0-41eb-9d89-26591d3e3ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs" Oct 30 01:16:46.710832 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.708150 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-host-run-multus-certs\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.710832 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.708217 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ecb9d3db-582f-41a0-945d-7bd7e4892ff8-serviceca\") pod \"node-ca-lm86p\" (UID: \"ecb9d3db-582f-41a0-945d-7bd7e4892ff8\") " pod="openshift-image-registry/node-ca-lm86p" Oct 30 01:16:46.710832 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:46.708238 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:16:46.710832 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.708270 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a2ec1edc-b911-4537-ac00-f708ef6c9062-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xfvmk\" (UID: \"a2ec1edc-b911-4537-ac00-f708ef6c9062\") " pod="openshift-multus/multus-additional-cni-plugins-xfvmk" Oct 30 01:16:46.710832 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.708279 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-etc-sysctl-d\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:16:46.710832 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.708344 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-host-var-lib-cni-bin\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.710832 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:46.708357 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94864d90-1dd8-472e-a414-29c6b6a4f8f6-metrics-certs podName:94864d90-1dd8-472e-a414-29c6b6a4f8f6 nodeName:}" failed. No retries permitted until 2026-10-30 01:16:47.208340296 +0000 UTC m=+2.104318258 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94864d90-1dd8-472e-a414-29c6b6a4f8f6-metrics-certs") pod "network-metrics-daemon-xj8mh" (UID: "94864d90-1dd8-472e-a414-29c6b6a4f8f6") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:16:46.710832 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.708365 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-etc-sysconfig\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:16:46.710832 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.708373 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a4365b4f-a9d0-41eb-9d89-26591d3e3ed5-device-dir\") pod \"aws-ebs-csi-driver-node-2wbfs\" (UID: \"a4365b4f-a9d0-41eb-9d89-26591d3e3ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs" Oct 30 01:16:46.711295 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.708394 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2ec1edc-b911-4537-ac00-f708ef6c9062-system-cni-dir\") pod \"multus-additional-cni-plugins-xfvmk\" (UID: \"a2ec1edc-b911-4537-ac00-f708ef6c9062\") " pod="openshift-multus/multus-additional-cni-plugins-xfvmk" Oct 30 01:16:46.711295 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.708378 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-host-run-netns\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.711295 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.708424 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-multus-conf-dir\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.711295 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.708519 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2ec1edc-b911-4537-ac00-f708ef6c9062-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xfvmk\" (UID: \"a2ec1edc-b911-4537-ac00-f708ef6c9062\") " pod="openshift-multus/multus-additional-cni-plugins-xfvmk" Oct 30 01:16:46.711295 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.709962 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e09d6f2e-7cd5-47c3-b18b-482726067e90-tmp\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:16:46.711295 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.710112 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e09d6f2e-7cd5-47c3-b18b-482726067e90-etc-tuned\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:16:46.711295 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.710222 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e30ba693-6284-485e-95fd-ad639627aa8d-agent-certs\") pod \"konnectivity-agent-9f9b4\" (UID: \"e30ba693-6284-485e-95fd-ad639627aa8d\") " pod="kube-system/konnectivity-agent-9f9b4" Oct 30 01:16:46.711295 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.710363 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d332bbb4-c699-44e8-96ba-516d8ca03e95-ovn-node-metrics-cert\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.727020 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:46.726999 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 01:16:46.727020 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:46.727023 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 01:16:46.727161 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:46.727057 2573 projected.go:196] Error preparing data for projected volume kube-api-access-89k2s for pod openshift-network-diagnostics/network-check-target-bbhx5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:16:46.727161 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:46.727105 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4602ac2-652b-4026-9a0d-d72162bb3258-kube-api-access-89k2s podName:f4602ac2-652b-4026-9a0d-d72162bb3258 nodeName:}" failed. No retries permitted until 2026-10-30 01:16:47.227090284 +0000 UTC m=+2.123068273 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-89k2s" (UniqueName: "kubernetes.io/projected/f4602ac2-652b-4026-9a0d-d72162bb3258-kube-api-access-89k2s") pod "network-check-target-bbhx5" (UID: "f4602ac2-652b-4026-9a0d-d72162bb3258") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:16:46.728967 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.728911 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rmpm\" (UniqueName: \"kubernetes.io/projected/94864d90-1dd8-472e-a414-29c6b6a4f8f6-kube-api-access-5rmpm\") pod \"network-metrics-daemon-xj8mh\" (UID: \"94864d90-1dd8-472e-a414-29c6b6a4f8f6\") " pod="openshift-multus/network-metrics-daemon-xj8mh" Oct 30 01:16:46.729277 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.729251 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qbvg\" (UniqueName: \"kubernetes.io/projected/a2ec1edc-b911-4537-ac00-f708ef6c9062-kube-api-access-8qbvg\") pod \"multus-additional-cni-plugins-xfvmk\" (UID: \"a2ec1edc-b911-4537-ac00-f708ef6c9062\") " pod="openshift-multus/multus-additional-cni-plugins-xfvmk" Oct 30 01:16:46.729990 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.729958 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w5t7\" (UniqueName: \"kubernetes.io/projected/d332bbb4-c699-44e8-96ba-516d8ca03e95-kube-api-access-8w5t7\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.730090 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.730010 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdvvv\" (UniqueName: \"kubernetes.io/projected/ecb9d3db-582f-41a0-945d-7bd7e4892ff8-kube-api-access-kdvvv\") pod \"node-ca-lm86p\" (UID: \"ecb9d3db-582f-41a0-945d-7bd7e4892ff8\") " pod="openshift-image-registry/node-ca-lm86p" Oct 30 01:16:46.730500 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.730480 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dhns\" (UniqueName: \"kubernetes.io/projected/c819c74a-0f8d-4a5e-9e1b-830196f6b117-kube-api-access-4dhns\") pod \"node-resolver-pf9jv\" (UID: \"c819c74a-0f8d-4a5e-9e1b-830196f6b117\") " pod="openshift-dns/node-resolver-pf9jv" Oct 30 01:16:46.730500 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.730493 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f67fc\" (UniqueName: \"kubernetes.io/projected/a4365b4f-a9d0-41eb-9d89-26591d3e3ed5-kube-api-access-f67fc\") pod \"aws-ebs-csi-driver-node-2wbfs\" (UID: \"a4365b4f-a9d0-41eb-9d89-26591d3e3ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs" Oct 30 01:16:46.730632 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.730613 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7pbf\" (UniqueName: \"kubernetes.io/projected/5618fce7-e2ca-4423-887a-eea73fdd9d01-kube-api-access-n7pbf\") pod \"iptables-alerter-mdwqw\" (UID: \"5618fce7-e2ca-4423-887a-eea73fdd9d01\") " pod="openshift-network-operator/iptables-alerter-mdwqw" Oct 30 01:16:46.730896 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.730879 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmptt\" (UniqueName: \"kubernetes.io/projected/e09d6f2e-7cd5-47c3-b18b-482726067e90-kube-api-access-nmptt\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:16:46.730975 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.730960 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5wrm\" (UniqueName: \"kubernetes.io/projected/bda81d83-62da-4f40-b668-29e2b0e51f35-kube-api-access-p5wrm\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.856139 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.856067 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pgk7v" Oct 30 01:16:46.865941 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:46.865917 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbda81d83_62da_4f40_b668_29e2b0e51f35.slice/crio-92b783d75cb547e19ac8d760f2932bea9da42a7227ad2efa48365245c4a84441 WatchSource:0}: Error finding container 92b783d75cb547e19ac8d760f2932bea9da42a7227ad2efa48365245c4a84441: Status 404 returned error can't find the container with id 92b783d75cb547e19ac8d760f2932bea9da42a7227ad2efa48365245c4a84441 Oct 30 01:16:46.883637 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.883618 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:16:46.890928 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.890911 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs" Oct 30 01:16:46.892378 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:46.892355 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd332bbb4_c699_44e8_96ba_516d8ca03e95.slice/crio-7e4a875aaf934ddc22726c19608ce2cbafc5a2da200c7f9949bd1d2c7d549a2d WatchSource:0}: Error finding container 7e4a875aaf934ddc22726c19608ce2cbafc5a2da200c7f9949bd1d2c7d549a2d: Status 404 returned error can't find the container with id 7e4a875aaf934ddc22726c19608ce2cbafc5a2da200c7f9949bd1d2c7d549a2d Oct 30 01:16:46.898480 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:46.898455 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4365b4f_a9d0_41eb_9d89_26591d3e3ed5.slice/crio-af72ee1cf15eaa41e0b953be6acb5ee16168b2df9406dd7cdb0f04a5e176ca02 WatchSource:0}: Error finding container af72ee1cf15eaa41e0b953be6acb5ee16168b2df9406dd7cdb0f04a5e176ca02: Status 404 returned error can't find the container with id af72ee1cf15eaa41e0b953be6acb5ee16168b2df9406dd7cdb0f04a5e176ca02 Oct 30 01:16:46.902662 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.902644 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pf9jv" Oct 30 01:16:46.912491 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:46.912462 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc819c74a_0f8d_4a5e_9e1b_830196f6b117.slice/crio-65640885d7180bd4cb191fa8b6d250f689ccfe4f63fb732180ae40bb71f3a4af WatchSource:0}: Error finding container 65640885d7180bd4cb191fa8b6d250f689ccfe4f63fb732180ae40bb71f3a4af: Status 404 returned error can't find the container with id 65640885d7180bd4cb191fa8b6d250f689ccfe4f63fb732180ae40bb71f3a4af Oct 30 01:16:46.913462 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.913436 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lm86p" Oct 30 01:16:46.922476 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:46.922456 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecb9d3db_582f_41a0_945d_7bd7e4892ff8.slice/crio-3eac419a77c7d5175c139a45a770985e2db87117baf8a3ce67ea4083e3e39ac6 WatchSource:0}: Error finding container 3eac419a77c7d5175c139a45a770985e2db87117baf8a3ce67ea4083e3e39ac6: Status 404 returned error can't find the container with id 3eac419a77c7d5175c139a45a770985e2db87117baf8a3ce67ea4083e3e39ac6 Oct 30 01:16:46.946020 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.946000 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xfvmk" Oct 30 01:16:46.954817 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:46.954796 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2ec1edc_b911_4537_ac00_f708ef6c9062.slice/crio-d69953189d01f4541102d21c4694ddde99fac0560b1983dda7d29603bd02ae87 WatchSource:0}: Error finding container d69953189d01f4541102d21c4694ddde99fac0560b1983dda7d29603bd02ae87: Status 404 returned error can't find the container with id d69953189d01f4541102d21c4694ddde99fac0560b1983dda7d29603bd02ae87 Oct 30 01:16:46.958024 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.958002 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mdwqw" Oct 30 01:16:46.961743 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.961727 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9f9b4" Oct 30 01:16:46.966211 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:46.966186 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5618fce7_e2ca_4423_887a_eea73fdd9d01.slice/crio-77762bb9330192855d2831289a47030363071dc9353470c76bece8a57c121c01 WatchSource:0}: Error finding container 77762bb9330192855d2831289a47030363071dc9353470c76bece8a57c121c01: Status 404 returned error can't find the container with id 77762bb9330192855d2831289a47030363071dc9353470c76bece8a57c121c01 Oct 30 01:16:46.966844 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:46.966828 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:16:46.970830 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:46.970809 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode30ba693_6284_485e_95fd_ad639627aa8d.slice/crio-f09ddf2fe29802b739251bcbfadbda34b9b134348512fc28b9f66fe7e9e5b497 WatchSource:0}: Error finding container f09ddf2fe29802b739251bcbfadbda34b9b134348512fc28b9f66fe7e9e5b497: Status 404 returned error can't find the container with id f09ddf2fe29802b739251bcbfadbda34b9b134348512fc28b9f66fe7e9e5b497 Oct 30 01:16:46.976355 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:16:46.976334 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode09d6f2e_7cd5_47c3_b18b_482726067e90.slice/crio-b48e7337e5520815153932ba59b809dc242f8e8dab1409a028e6502c6d5add6b WatchSource:0}: Error finding container b48e7337e5520815153932ba59b809dc242f8e8dab1409a028e6502c6d5add6b: Status 404 returned error can't find the container with id b48e7337e5520815153932ba59b809dc242f8e8dab1409a028e6502c6d5add6b Oct 30 01:16:47.212164 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:47.212089 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94864d90-1dd8-472e-a414-29c6b6a4f8f6-metrics-certs\") pod \"network-metrics-daemon-xj8mh\" (UID: \"94864d90-1dd8-472e-a414-29c6b6a4f8f6\") " pod="openshift-multus/network-metrics-daemon-xj8mh" Oct 30 01:16:47.212319 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:47.212161 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f10aab9c-d4e6-45b5-a703-146114a31329-original-pull-secret\") pod \"global-pull-secret-syncer-dxz45\" (UID: \"f10aab9c-d4e6-45b5-a703-146114a31329\") " pod="kube-system/global-pull-secret-syncer-dxz45" Oct 30 01:16:47.212319 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:47.212294 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 01:16:47.212425 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:47.212365 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f10aab9c-d4e6-45b5-a703-146114a31329-original-pull-secret podName:f10aab9c-d4e6-45b5-a703-146114a31329 nodeName:}" failed. No retries permitted until 2026-10-30 01:16:48.212347556 +0000 UTC m=+3.108325522 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f10aab9c-d4e6-45b5-a703-146114a31329-original-pull-secret") pod "global-pull-secret-syncer-dxz45" (UID: "f10aab9c-d4e6-45b5-a703-146114a31329") : object "kube-system"/"original-pull-secret" not registered Oct 30 01:16:47.212789 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:47.212764 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:16:47.212892 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:47.212819 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94864d90-1dd8-472e-a414-29c6b6a4f8f6-metrics-certs podName:94864d90-1dd8-472e-a414-29c6b6a4f8f6 nodeName:}" failed. No retries permitted until 2026-10-30 01:16:48.21280408 +0000 UTC m=+3.108782047 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94864d90-1dd8-472e-a414-29c6b6a4f8f6-metrics-certs") pod "network-metrics-daemon-xj8mh" (UID: "94864d90-1dd8-472e-a414-29c6b6a4f8f6") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:16:47.313287 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:47.313247 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89k2s\" (UniqueName: \"kubernetes.io/projected/f4602ac2-652b-4026-9a0d-d72162bb3258-kube-api-access-89k2s\") pod \"network-check-target-bbhx5\" (UID: \"f4602ac2-652b-4026-9a0d-d72162bb3258\") " pod="openshift-network-diagnostics/network-check-target-bbhx5" Oct 30 01:16:47.313440 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:47.313418 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 01:16:47.313440 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:47.313437 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 01:16:47.313574 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:47.313459 2573 projected.go:196] Error preparing data for projected volume kube-api-access-89k2s for pod openshift-network-diagnostics/network-check-target-bbhx5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:16:47.313574 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:47.313510 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4602ac2-652b-4026-9a0d-d72162bb3258-kube-api-access-89k2s podName:f4602ac2-652b-4026-9a0d-d72162bb3258 nodeName:}" failed. No retries permitted until 2026-10-30 01:16:48.313496716 +0000 UTC m=+3.209474677 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-89k2s" (UniqueName: "kubernetes.io/projected/f4602ac2-652b-4026-9a0d-d72162bb3258-kube-api-access-89k2s") pod "network-check-target-bbhx5" (UID: "f4602ac2-652b-4026-9a0d-d72162bb3258") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:16:47.415444 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:47.415406 2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:16:47.557456 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:47.557302 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 01:11:46 +0000 UTC" deadline="2028-04-18 16:54:25.797007361 +0000 UTC" Oct 30 01:16:47.557456 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:47.557357 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12879h37m38.239654531s" Oct 30 01:16:47.623873 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:47.623340 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbhx5" Oct 30 01:16:47.623873 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:47.623477 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbhx5" podUID="f4602ac2-652b-4026-9a0d-d72162bb3258" Oct 30 01:16:47.650962 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:47.650851 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" event={"ID":"e09d6f2e-7cd5-47c3-b18b-482726067e90","Type":"ContainerStarted","Data":"b48e7337e5520815153932ba59b809dc242f8e8dab1409a028e6502c6d5add6b"} Oct 30 01:16:47.655186 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:47.655155 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mdwqw" event={"ID":"5618fce7-e2ca-4423-887a-eea73fdd9d01","Type":"ContainerStarted","Data":"77762bb9330192855d2831289a47030363071dc9353470c76bece8a57c121c01"} Oct 30 01:16:47.662253 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:47.662224 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xfvmk" event={"ID":"a2ec1edc-b911-4537-ac00-f708ef6c9062","Type":"ContainerStarted","Data":"d69953189d01f4541102d21c4694ddde99fac0560b1983dda7d29603bd02ae87"} Oct 30 01:16:47.665861 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:47.665679 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lm86p" event={"ID":"ecb9d3db-582f-41a0-945d-7bd7e4892ff8","Type":"ContainerStarted","Data":"3eac419a77c7d5175c139a45a770985e2db87117baf8a3ce67ea4083e3e39ac6"} Oct 30 01:16:47.685992 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:47.685958 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs" event={"ID":"a4365b4f-a9d0-41eb-9d89-26591d3e3ed5","Type":"ContainerStarted","Data":"af72ee1cf15eaa41e0b953be6acb5ee16168b2df9406dd7cdb0f04a5e176ca02"} Oct 30 01:16:47.701250 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:47.701149 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" event={"ID":"d332bbb4-c699-44e8-96ba-516d8ca03e95","Type":"ContainerStarted","Data":"7e4a875aaf934ddc22726c19608ce2cbafc5a2da200c7f9949bd1d2c7d549a2d"} Oct 30 01:16:47.707411 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:47.707326 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-65.ec2.internal" event={"ID":"7af5b4d87431a44e4b7050910f6ad38b","Type":"ContainerStarted","Data":"415e62159ff51389db9ac1f6f638080802e928cef05b934bed5ae6c9b0e64520"} Oct 30 01:16:47.712376 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:47.712298 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-65.ec2.internal" event={"ID":"a707a897ca34d8ee2c04b93cf900a7fe","Type":"ContainerStarted","Data":"e357503c9c89fb4c2823d4997882a3185c18d2abb7aa618b6b31f657e077742d"} Oct 30 01:16:47.727402 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:47.727321 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9f9b4" event={"ID":"e30ba693-6284-485e-95fd-ad639627aa8d","Type":"ContainerStarted","Data":"f09ddf2fe29802b739251bcbfadbda34b9b134348512fc28b9f66fe7e9e5b497"} Oct 30 01:16:47.738752 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:47.738725 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pf9jv" event={"ID":"c819c74a-0f8d-4a5e-9e1b-830196f6b117","Type":"ContainerStarted","Data":"65640885d7180bd4cb191fa8b6d250f689ccfe4f63fb732180ae40bb71f3a4af"} Oct 30 01:16:47.742649 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:47.742593 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pgk7v" event={"ID":"bda81d83-62da-4f40-b668-29e2b0e51f35","Type":"ContainerStarted","Data":"92b783d75cb547e19ac8d760f2932bea9da42a7227ad2efa48365245c4a84441"} Oct 30 01:16:47.864817 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:47.864719 2573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:16:47.995502 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:47.995471 2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:16:48.222501 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:48.221572 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f10aab9c-d4e6-45b5-a703-146114a31329-original-pull-secret\") pod \"global-pull-secret-syncer-dxz45\" (UID: \"f10aab9c-d4e6-45b5-a703-146114a31329\") " pod="kube-system/global-pull-secret-syncer-dxz45" Oct 30 01:16:48.222501 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:48.221667 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94864d90-1dd8-472e-a414-29c6b6a4f8f6-metrics-certs\") pod \"network-metrics-daemon-xj8mh\" (UID: \"94864d90-1dd8-472e-a414-29c6b6a4f8f6\") " pod="openshift-multus/network-metrics-daemon-xj8mh" Oct 30 01:16:48.222501 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:48.221831 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:16:48.222501 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:48.221931 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94864d90-1dd8-472e-a414-29c6b6a4f8f6-metrics-certs podName:94864d90-1dd8-472e-a414-29c6b6a4f8f6 nodeName:}" failed. No retries permitted until 2026-10-30 01:16:50.22191366 +0000 UTC m=+5.117891638 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94864d90-1dd8-472e-a414-29c6b6a4f8f6-metrics-certs") pod "network-metrics-daemon-xj8mh" (UID: "94864d90-1dd8-472e-a414-29c6b6a4f8f6") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:16:48.222501 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:48.222380 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 01:16:48.222501 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:48.222429 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f10aab9c-d4e6-45b5-a703-146114a31329-original-pull-secret podName:f10aab9c-d4e6-45b5-a703-146114a31329 nodeName:}" failed. No retries permitted until 2026-10-30 01:16:50.22241336 +0000 UTC m=+5.118391326 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f10aab9c-d4e6-45b5-a703-146114a31329-original-pull-secret") pod "global-pull-secret-syncer-dxz45" (UID: "f10aab9c-d4e6-45b5-a703-146114a31329") : object "kube-system"/"original-pull-secret" not registered Oct 30 01:16:48.323050 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:48.323015 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89k2s\" (UniqueName: \"kubernetes.io/projected/f4602ac2-652b-4026-9a0d-d72162bb3258-kube-api-access-89k2s\") pod \"network-check-target-bbhx5\" (UID: \"f4602ac2-652b-4026-9a0d-d72162bb3258\") " pod="openshift-network-diagnostics/network-check-target-bbhx5" Oct 30 01:16:48.323226 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:48.323176 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 01:16:48.323226 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:48.323200 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 01:16:48.323226 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:48.323226 2573 projected.go:196] Error preparing data for projected volume kube-api-access-89k2s for pod openshift-network-diagnostics/network-check-target-bbhx5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:16:48.323374 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:48.323290 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4602ac2-652b-4026-9a0d-d72162bb3258-kube-api-access-89k2s podName:f4602ac2-652b-4026-9a0d-d72162bb3258 nodeName:}" failed. No retries permitted until 2026-10-30 01:16:50.323273171 +0000 UTC m=+5.219251138 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-89k2s" (UniqueName: "kubernetes.io/projected/f4602ac2-652b-4026-9a0d-d72162bb3258-kube-api-access-89k2s") pod "network-check-target-bbhx5" (UID: "f4602ac2-652b-4026-9a0d-d72162bb3258") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:16:48.558167 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:48.557993 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 01:11:46 +0000 UTC" deadline="2028-04-03 05:25:26.273651436 +0000 UTC" Oct 30 01:16:48.558167 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:48.558030 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12508h8m37.715625661s" Oct 30 01:16:48.621015 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:48.620978 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xj8mh" Oct 30 01:16:48.621202 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:48.621127 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xj8mh" podUID="94864d90-1dd8-472e-a414-29c6b6a4f8f6" Oct 30 01:16:48.621800 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:48.621551 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxz45" Oct 30 01:16:48.621800 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:48.621648 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxz45" podUID="f10aab9c-d4e6-45b5-a703-146114a31329" Oct 30 01:16:49.621395 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:49.620916 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbhx5" Oct 30 01:16:49.621395 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:49.621056 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbhx5" podUID="f4602ac2-652b-4026-9a0d-d72162bb3258" Oct 30 01:16:50.243915 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:50.243878 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f10aab9c-d4e6-45b5-a703-146114a31329-original-pull-secret\") pod \"global-pull-secret-syncer-dxz45\" (UID: \"f10aab9c-d4e6-45b5-a703-146114a31329\") " pod="kube-system/global-pull-secret-syncer-dxz45" Oct 30 01:16:50.244037 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:50.243961 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94864d90-1dd8-472e-a414-29c6b6a4f8f6-metrics-certs\") pod \"network-metrics-daemon-xj8mh\" (UID: \"94864d90-1dd8-472e-a414-29c6b6a4f8f6\") " pod="openshift-multus/network-metrics-daemon-xj8mh" Oct 30 01:16:50.244037 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:50.244024 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 01:16:50.244157 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:50.244043 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:16:50.244157 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:50.244084 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94864d90-1dd8-472e-a414-29c6b6a4f8f6-metrics-certs podName:94864d90-1dd8-472e-a414-29c6b6a4f8f6 nodeName:}" failed. No retries permitted until 2026-10-30 01:16:54.244067471 +0000 UTC m=+9.140045447 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94864d90-1dd8-472e-a414-29c6b6a4f8f6-metrics-certs") pod "network-metrics-daemon-xj8mh" (UID: "94864d90-1dd8-472e-a414-29c6b6a4f8f6") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:16:50.244157 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:50.244100 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f10aab9c-d4e6-45b5-a703-146114a31329-original-pull-secret podName:f10aab9c-d4e6-45b5-a703-146114a31329 nodeName:}" failed. No retries permitted until 2026-10-30 01:16:54.244091816 +0000 UTC m=+9.140069777 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f10aab9c-d4e6-45b5-a703-146114a31329-original-pull-secret") pod "global-pull-secret-syncer-dxz45" (UID: "f10aab9c-d4e6-45b5-a703-146114a31329") : object "kube-system"/"original-pull-secret" not registered Oct 30 01:16:50.345267 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:50.345082 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89k2s\" (UniqueName: \"kubernetes.io/projected/f4602ac2-652b-4026-9a0d-d72162bb3258-kube-api-access-89k2s\") pod \"network-check-target-bbhx5\" (UID: \"f4602ac2-652b-4026-9a0d-d72162bb3258\") " pod="openshift-network-diagnostics/network-check-target-bbhx5" Oct 30 01:16:50.345496 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:50.345399 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 01:16:50.345496 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:50.345426 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 01:16:50.345496 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:50.345438 2573 projected.go:196] Error preparing data for projected volume kube-api-access-89k2s for pod openshift-network-diagnostics/network-check-target-bbhx5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:16:50.345672 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:50.345503 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4602ac2-652b-4026-9a0d-d72162bb3258-kube-api-access-89k2s podName:f4602ac2-652b-4026-9a0d-d72162bb3258 nodeName:}" failed. No retries permitted until 2026-10-30 01:16:54.345483815 +0000 UTC m=+9.241461790 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-89k2s" (UniqueName: "kubernetes.io/projected/f4602ac2-652b-4026-9a0d-d72162bb3258-kube-api-access-89k2s") pod "network-check-target-bbhx5" (UID: "f4602ac2-652b-4026-9a0d-d72162bb3258") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:16:50.621909 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:50.621163 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xj8mh" Oct 30 01:16:50.621909 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:50.621314 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xj8mh" podUID="94864d90-1dd8-472e-a414-29c6b6a4f8f6" Oct 30 01:16:50.621909 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:50.621773 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxz45" Oct 30 01:16:50.621909 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:50.621867 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxz45" podUID="f10aab9c-d4e6-45b5-a703-146114a31329" Oct 30 01:16:50.774093 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:50.773125 2573 generic.go:356] "Generic (PLEG): container finished" podID="a707a897ca34d8ee2c04b93cf900a7fe" containerID="f650c53e0fbc5e060c63f8fe0b1a3717f4011b1c615b62d79cfb747b475ff9db" exitCode=0 Oct 30 01:16:50.774093 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:50.773228 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-65.ec2.internal" event={"ID":"a707a897ca34d8ee2c04b93cf900a7fe","Type":"ContainerDied","Data":"f650c53e0fbc5e060c63f8fe0b1a3717f4011b1c615b62d79cfb747b475ff9db"} Oct 30 01:16:50.790108 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:50.788523 2573 generic.go:356] "Generic (PLEG): container finished" podID="a2ec1edc-b911-4537-ac00-f708ef6c9062" containerID="0cf075b35ef887db402a9fc1eb9f571aa9debf1d1350ead6acd56e93d573d921" exitCode=0 Oct 30 01:16:50.790108 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:50.788577 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xfvmk" event={"ID":"a2ec1edc-b911-4537-ac00-f708ef6c9062","Type":"ContainerDied","Data":"0cf075b35ef887db402a9fc1eb9f571aa9debf1d1350ead6acd56e93d573d921"} Oct 30 01:16:51.621420 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:51.621388 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbhx5" Oct 30 01:16:51.621633 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:51.621511 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbhx5" podUID="f4602ac2-652b-4026-9a0d-d72162bb3258" Oct 30 01:16:51.794739 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:51.794697 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-65.ec2.internal" event={"ID":"7af5b4d87431a44e4b7050910f6ad38b","Type":"ContainerStarted","Data":"24c74d59abeef83fa778f4eda6f5b24e11272bf67efc350f917a8284348edfac"} Oct 30 01:16:51.800920 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:51.800887 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-65.ec2.internal" event={"ID":"a707a897ca34d8ee2c04b93cf900a7fe","Type":"ContainerStarted","Data":"d36d752ffc2bfa45f23cdac39293fef6b23fee8e9a1f7aa39f0afe1d235a1e0f"} Oct 30 01:16:51.805081 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:51.805052 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9f9b4" event={"ID":"e30ba693-6284-485e-95fd-ad639627aa8d","Type":"ContainerStarted","Data":"2c3aabd8a942b5fd140e9ea9a9e5789136e9e1a4b0d12f6a71cc898fc7252b41"} Oct 30 01:16:51.873066 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:51.872435 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-65.ec2.internal" podStartSLOduration=6.872415143 podStartE2EDuration="6.872415143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:16:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:51.832717434 +0000 UTC m=+6.728695419" watchObservedRunningTime="2026-10-30 01:16:51.872415143 +0000 UTC m=+6.768393130" Oct 30 01:16:51.913107 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:51.912168 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9f9b4" podStartSLOduration=2.6322697010000002 podStartE2EDuration="6.912149501s" totalImagesPullingTime="4.2798798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:51.911633103 +0000 UTC m=+6.807611088" watchObservedRunningTime="2026-10-30 01:16:51.912149501 +0000 UTC m=+6.808127485" Oct 30 01:16:51.913107 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:51.912812 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-65.ec2.internal" podStartSLOduration=6.912798274 podStartE2EDuration="6.912798274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:16:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:51.872927398 +0000 UTC m=+6.768905383" watchObservedRunningTime="2026-10-30 01:16:51.912798274 +0000 UTC m=+6.808776258" Oct 30 01:16:52.621433 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:52.621404 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xj8mh" Oct 30 01:16:52.621654 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:52.621582 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xj8mh" podUID="94864d90-1dd8-472e-a414-29c6b6a4f8f6" Oct 30 01:16:52.622114 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:52.622046 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxz45" Oct 30 01:16:52.622203 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:52.622146 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxz45" podUID="f10aab9c-d4e6-45b5-a703-146114a31329" Oct 30 01:16:52.735723 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:52.735643 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9f9b4" Oct 30 01:16:52.736317 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:52.736285 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9f9b4" Oct 30 01:16:52.815096 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:52.814868 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lm86p" event={"ID":"ecb9d3db-582f-41a0-945d-7bd7e4892ff8","Type":"ContainerStarted","Data":"b4bc7997bce051f4088d5f613858a7e52add05e0177079a7014f4435b5b2d283"} Oct 30 01:16:52.815096 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:52.814922 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9f9b4" Oct 30 01:16:52.816324 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:52.816300 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9f9b4" Oct 30 01:16:52.898294 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:52.897247 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lm86p" podStartSLOduration=2.352670759 podStartE2EDuration="7.897227221s" totalImagesPullingTime="5.544556462s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:52.854629179 +0000 UTC m=+7.750607163" watchObservedRunningTime="2026-10-30 01:16:52.897227221 +0000 UTC m=+7.793205205" Oct 30 01:16:53.621794 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:53.621750 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbhx5" Oct 30 01:16:53.621970 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:53.621898 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbhx5" podUID="f4602ac2-652b-4026-9a0d-d72162bb3258" Oct 30 01:16:53.819724 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:53.819692 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs" event={"ID":"a4365b4f-a9d0-41eb-9d89-26591d3e3ed5","Type":"ContainerStarted","Data":"e1c70c9c0ba2fb3eb173bcf472335d9bc02f66495283f7468f1dec1e93082cf1"} Oct 30 01:16:54.279460 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:54.279418 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94864d90-1dd8-472e-a414-29c6b6a4f8f6-metrics-certs\") pod \"network-metrics-daemon-xj8mh\" (UID: \"94864d90-1dd8-472e-a414-29c6b6a4f8f6\") " pod="openshift-multus/network-metrics-daemon-xj8mh" Oct 30 01:16:54.279649 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:54.279498 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f10aab9c-d4e6-45b5-a703-146114a31329-original-pull-secret\") pod \"global-pull-secret-syncer-dxz45\" (UID: \"f10aab9c-d4e6-45b5-a703-146114a31329\") " pod="kube-system/global-pull-secret-syncer-dxz45" Oct 30 01:16:54.279712 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:54.279663 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 01:16:54.279755 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:54.279727 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f10aab9c-d4e6-45b5-a703-146114a31329-original-pull-secret podName:f10aab9c-d4e6-45b5-a703-146114a31329 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:02.279710865 +0000 UTC m=+17.175688831 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f10aab9c-d4e6-45b5-a703-146114a31329-original-pull-secret") pod "global-pull-secret-syncer-dxz45" (UID: "f10aab9c-d4e6-45b5-a703-146114a31329") : object "kube-system"/"original-pull-secret" not registered Oct 30 01:16:54.280214 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:54.280195 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:16:54.280307 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:54.280251 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94864d90-1dd8-472e-a414-29c6b6a4f8f6-metrics-certs podName:94864d90-1dd8-472e-a414-29c6b6a4f8f6 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:02.280236108 +0000 UTC m=+17.176214075 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94864d90-1dd8-472e-a414-29c6b6a4f8f6-metrics-certs") pod "network-metrics-daemon-xj8mh" (UID: "94864d90-1dd8-472e-a414-29c6b6a4f8f6") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:16:54.380421 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:54.380384 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89k2s\" (UniqueName: \"kubernetes.io/projected/f4602ac2-652b-4026-9a0d-d72162bb3258-kube-api-access-89k2s\") pod \"network-check-target-bbhx5\" (UID: \"f4602ac2-652b-4026-9a0d-d72162bb3258\") " pod="openshift-network-diagnostics/network-check-target-bbhx5" Oct 30 01:16:54.380617 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:54.380596 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 01:16:54.380688 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:54.380623 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 01:16:54.380688 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:54.380637 2573 projected.go:196] Error preparing data for projected volume kube-api-access-89k2s for pod openshift-network-diagnostics/network-check-target-bbhx5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:16:54.380762 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:54.380701 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4602ac2-652b-4026-9a0d-d72162bb3258-kube-api-access-89k2s podName:f4602ac2-652b-4026-9a0d-d72162bb3258 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:02.380682167 +0000 UTC m=+17.276660148 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-89k2s" (UniqueName: "kubernetes.io/projected/f4602ac2-652b-4026-9a0d-d72162bb3258-kube-api-access-89k2s") pod "network-check-target-bbhx5" (UID: "f4602ac2-652b-4026-9a0d-d72162bb3258") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:16:54.620823 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:54.620790 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xj8mh" Oct 30 01:16:54.620988 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:54.620934 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xj8mh" podUID="94864d90-1dd8-472e-a414-29c6b6a4f8f6" Oct 30 01:16:54.621342 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:54.621321 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxz45" Oct 30 01:16:54.621454 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:54.621420 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxz45" podUID="f10aab9c-d4e6-45b5-a703-146114a31329" Oct 30 01:16:54.942667 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:54.942606 2573 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 01:16:55.563387 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:55.563291 2573 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T01:16:54.942626689Z","UUID":"c83e5d47-8375-498d-af10-8ce9cb8f517d","Handler":null,"Name":"","Endpoint":""} Oct 30 01:16:55.565357 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:55.565206 2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 01:16:55.565357 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:55.565227 2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 01:16:55.621396 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:55.621361 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbhx5" Oct 30 01:16:55.621603 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:55.621462 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbhx5" podUID="f4602ac2-652b-4026-9a0d-d72162bb3258" Oct 30 01:16:55.826025 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:55.825402 2573 generic.go:356] "Generic (PLEG): container finished" podID="a2ec1edc-b911-4537-ac00-f708ef6c9062" containerID="039acf50b948637d60e6123a560a9feeb4775d2977aeddf9fd23a3a1cf164c16" exitCode=0 Oct 30 01:16:55.826025 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:55.825500 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xfvmk" event={"ID":"a2ec1edc-b911-4537-ac00-f708ef6c9062","Type":"ContainerDied","Data":"039acf50b948637d60e6123a560a9feeb4775d2977aeddf9fd23a3a1cf164c16"} Oct 30 01:16:55.827716 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:55.827687 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs" event={"ID":"a4365b4f-a9d0-41eb-9d89-26591d3e3ed5","Type":"ContainerStarted","Data":"ecf61bb227af002171a218690eb588ccea84c0eef9e6829941449cc8703af2f9"} Oct 30 01:16:55.829605 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:55.829576 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pf9jv" event={"ID":"c819c74a-0f8d-4a5e-9e1b-830196f6b117","Type":"ContainerStarted","Data":"96b50bbc661e480e7b0813acc0770de64bd9768d987a4194c957cc44dbb06176"} Oct 30 01:16:55.831943 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:55.831912 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pgk7v" event={"ID":"bda81d83-62da-4f40-b668-29e2b0e51f35","Type":"ContainerStarted","Data":"9c454f117e73b6907b8c2d23580c13f38c802af30b9d1d8debe750ad4ff52ba4"} Oct 30 01:16:55.948761 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:55.948577 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-pgk7v" podStartSLOduration=2.80288028 podStartE2EDuration="10.948562918s" totalImagesPullingTime="8.145682638s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:55.91024131 +0000 UTC m=+10.806219318" watchObservedRunningTime="2026-10-30 01:16:55.948562918 +0000 UTC m=+10.844540900" Oct 30 01:16:56.621390 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:56.621361 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xj8mh" Oct 30 01:16:56.621562 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:56.621497 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xj8mh" podUID="94864d90-1dd8-472e-a414-29c6b6a4f8f6" Oct 30 01:16:56.621832 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:56.621817 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxz45" Oct 30 01:16:56.621913 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:56.621897 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxz45" podUID="f10aab9c-d4e6-45b5-a703-146114a31329" Oct 30 01:16:56.837073 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:56.837039 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs" event={"ID":"a4365b4f-a9d0-41eb-9d89-26591d3e3ed5","Type":"ContainerStarted","Data":"541229c5e989b2b5b23e4ce2b724f7307d9bdd930763c974f56c8c06a80a7fa1"} Oct 30 01:16:56.839840 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:56.839374 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mdwqw" event={"ID":"5618fce7-e2ca-4423-887a-eea73fdd9d01","Type":"ContainerStarted","Data":"62324156ea98ed48d9592ecc317565011a6b825fecfaa1a69f598bfed47bbac1"} Oct 30 01:16:56.842185 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:56.842155 2573 generic.go:356] "Generic (PLEG): container finished" podID="a2ec1edc-b911-4537-ac00-f708ef6c9062" containerID="24f1c9127a58360755dca345cc6330058a4b89b515cb75b47698047f6165dfb1" exitCode=0 Oct 30 01:16:56.842297 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:56.842259 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xfvmk" event={"ID":"a2ec1edc-b911-4537-ac00-f708ef6c9062","Type":"ContainerDied","Data":"24f1c9127a58360755dca345cc6330058a4b89b515cb75b47698047f6165dfb1"} Oct 30 01:16:56.879127 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:56.879081 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-pf9jv" podStartSLOduration=3.797249594 podStartE2EDuration="11.879068415s" totalImagesPullingTime="8.081818821s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:55.948843488 +0000 UTC m=+10.844821471" watchObservedRunningTime="2026-10-30 01:16:56.879068415 +0000 UTC m=+11.775046398" Oct 30 01:16:56.927048 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:56.926983 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs" podStartSLOduration=3.37084807 podStartE2EDuration="11.926965664s" totalImagesPullingTime="8.556117594s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:16:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:56.879544871 +0000 UTC m=+11.775522850" watchObservedRunningTime="2026-10-30 01:16:56.926965664 +0000 UTC m=+11.822943648" Oct 30 01:16:56.966653 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:56.966587 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-mdwqw" podStartSLOduration=3.650731093 podStartE2EDuration="11.96657032s" totalImagesPullingTime="8.315839227s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:56.965851613 +0000 UTC m=+11.861829597" watchObservedRunningTime="2026-10-30 01:16:56.96657032 +0000 UTC m=+11.862548305" Oct 30 01:16:57.620697 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:57.620622 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbhx5" Oct 30 01:16:57.620842 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:57.620750 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbhx5" podUID="f4602ac2-652b-4026-9a0d-d72162bb3258" Oct 30 01:16:57.845844 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:57.845802 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" event={"ID":"e09d6f2e-7cd5-47c3-b18b-482726067e90","Type":"ContainerStarted","Data":"59631b1d13d8f720b6f120b8835bcd54d40da377bac1c71d9fbdcf88c493dd97"} Oct 30 01:16:57.848549 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:57.848499 2573 generic.go:356] "Generic (PLEG): container finished" podID="a2ec1edc-b911-4537-ac00-f708ef6c9062" containerID="727d892a7f2f45f72d36529dfc6d7e79d888779eee6fce1084b3d43c77bdeb7c" exitCode=0 Oct 30 01:16:57.848680 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:57.848585 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xfvmk" event={"ID":"a2ec1edc-b911-4537-ac00-f708ef6c9062","Type":"ContainerDied","Data":"727d892a7f2f45f72d36529dfc6d7e79d888779eee6fce1084b3d43c77bdeb7c"} Oct 30 01:16:57.906896 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:57.906793 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" podStartSLOduration=2.518990889 podStartE2EDuration="12.906774269s" totalImagesPullingTime="10.38778338s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:16:57.905130119 +0000 UTC m=+12.801108104" watchObservedRunningTime="2026-10-30 01:16:57.906774269 +0000 UTC m=+12.802752253" Oct 30 01:16:58.620906 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:58.620524 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xj8mh" Oct 30 01:16:58.620906 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:58.620583 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxz45" Oct 30 01:16:58.620906 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:58.620681 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xj8mh" podUID="94864d90-1dd8-472e-a414-29c6b6a4f8f6" Oct 30 01:16:58.620906 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:58.620818 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxz45" podUID="f10aab9c-d4e6-45b5-a703-146114a31329" Oct 30 01:16:59.621286 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:59.621254 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbhx5" Oct 30 01:16:59.621814 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:16:59.621387 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbhx5" podUID="f4602ac2-652b-4026-9a0d-d72162bb3258" Oct 30 01:16:59.855904 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:16:59.855444 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xfvmk" event={"ID":"a2ec1edc-b911-4537-ac00-f708ef6c9062","Type":"ContainerStarted","Data":"4b2a2aae230e0151dc4b9d8ff9bcfd3f70c67b08a1050ad1bd1607d51c156785"} Oct 30 01:17:00.621287 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:00.621193 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxz45" Oct 30 01:17:00.621287 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:00.621238 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xj8mh" Oct 30 01:17:00.621875 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:00.621340 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxz45" podUID="f10aab9c-d4e6-45b5-a703-146114a31329" Oct 30 01:17:00.621875 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:00.621672 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xj8mh" podUID="94864d90-1dd8-472e-a414-29c6b6a4f8f6" Oct 30 01:17:00.861647 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:00.861613 2573 generic.go:356] "Generic (PLEG): container finished" podID="a2ec1edc-b911-4537-ac00-f708ef6c9062" containerID="4b2a2aae230e0151dc4b9d8ff9bcfd3f70c67b08a1050ad1bd1607d51c156785" exitCode=0 Oct 30 01:17:00.861805 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:00.861669 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xfvmk" event={"ID":"a2ec1edc-b911-4537-ac00-f708ef6c9062","Type":"ContainerDied","Data":"4b2a2aae230e0151dc4b9d8ff9bcfd3f70c67b08a1050ad1bd1607d51c156785"} Oct 30 01:17:01.620475 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:01.620453 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbhx5" Oct 30 01:17:01.620588 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:01.620565 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbhx5" podUID="f4602ac2-652b-4026-9a0d-d72162bb3258" Oct 30 01:17:01.867254 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:01.867214 2573 generic.go:356] "Generic (PLEG): container finished" podID="a2ec1edc-b911-4537-ac00-f708ef6c9062" containerID="370e8a1f6e0cbe68cc5e92bc862b06aef94633f5234e2492c490210f9b65c831" exitCode=0 Oct 30 01:17:01.867631 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:01.867301 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xfvmk" event={"ID":"a2ec1edc-b911-4537-ac00-f708ef6c9062","Type":"ContainerDied","Data":"370e8a1f6e0cbe68cc5e92bc862b06aef94633f5234e2492c490210f9b65c831"} Oct 30 01:17:01.869214 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:01.869192 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2vsns_d332bbb4-c699-44e8-96ba-516d8ca03e95/ovn-acl-logging/0.log" Oct 30 01:17:01.869552 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:01.869516 2573 generic.go:356] "Generic (PLEG): container finished" podID="d332bbb4-c699-44e8-96ba-516d8ca03e95" containerID="c9e1e3083fc57e5d20d155273f7f436856af4d4f9acdf10b090e03b8c4a2a826" exitCode=1 Oct 30 01:17:01.869606 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:01.869557 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" event={"ID":"d332bbb4-c699-44e8-96ba-516d8ca03e95","Type":"ContainerStarted","Data":"1d3198b57603e0ad2cfb15ac824c6f78407f78997fad65f61994e3d361db9bde"} Oct 30 01:17:01.869646 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:01.869618 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" event={"ID":"d332bbb4-c699-44e8-96ba-516d8ca03e95","Type":"ContainerDied","Data":"c9e1e3083fc57e5d20d155273f7f436856af4d4f9acdf10b090e03b8c4a2a826"} Oct 30 01:17:01.869677 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:01.869655 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" event={"ID":"d332bbb4-c699-44e8-96ba-516d8ca03e95","Type":"ContainerStarted","Data":"34c74b546db717e92379e4e9e1bee9cceb2e2d91588b4c57e8c2a5aa636efa0e"} Oct 30 01:17:02.344046 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:02.344004 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94864d90-1dd8-472e-a414-29c6b6a4f8f6-metrics-certs\") pod \"network-metrics-daemon-xj8mh\" (UID: \"94864d90-1dd8-472e-a414-29c6b6a4f8f6\") " pod="openshift-multus/network-metrics-daemon-xj8mh" Oct 30 01:17:02.344355 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:02.344058 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f10aab9c-d4e6-45b5-a703-146114a31329-original-pull-secret\") pod \"global-pull-secret-syncer-dxz45\" (UID: \"f10aab9c-d4e6-45b5-a703-146114a31329\") " pod="kube-system/global-pull-secret-syncer-dxz45" Oct 30 01:17:02.344355 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:02.344147 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 01:17:02.344355 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:02.344152 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:17:02.344355 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:02.344198 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f10aab9c-d4e6-45b5-a703-146114a31329-original-pull-secret podName:f10aab9c-d4e6-45b5-a703-146114a31329 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:18.344186063 +0000 UTC m=+33.240164024 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f10aab9c-d4e6-45b5-a703-146114a31329-original-pull-secret") pod "global-pull-secret-syncer-dxz45" (UID: "f10aab9c-d4e6-45b5-a703-146114a31329") : object "kube-system"/"original-pull-secret" not registered Oct 30 01:17:02.344355 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:02.344212 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94864d90-1dd8-472e-a414-29c6b6a4f8f6-metrics-certs podName:94864d90-1dd8-472e-a414-29c6b6a4f8f6 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:18.344206076 +0000 UTC m=+33.240184036 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94864d90-1dd8-472e-a414-29c6b6a4f8f6-metrics-certs") pod "network-metrics-daemon-xj8mh" (UID: "94864d90-1dd8-472e-a414-29c6b6a4f8f6") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:17:02.444903 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:02.444861 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89k2s\" (UniqueName: \"kubernetes.io/projected/f4602ac2-652b-4026-9a0d-d72162bb3258-kube-api-access-89k2s\") pod \"network-check-target-bbhx5\" (UID: \"f4602ac2-652b-4026-9a0d-d72162bb3258\") " pod="openshift-network-diagnostics/network-check-target-bbhx5" Oct 30 01:17:02.445064 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:02.445036 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 01:17:02.445064 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:02.445062 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 01:17:02.445134 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:02.445074 2573 projected.go:196] Error preparing data for projected volume kube-api-access-89k2s for pod openshift-network-diagnostics/network-check-target-bbhx5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:17:02.445134 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:02.445122 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4602ac2-652b-4026-9a0d-d72162bb3258-kube-api-access-89k2s podName:f4602ac2-652b-4026-9a0d-d72162bb3258 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:18.445110236 +0000 UTC m=+33.341088202 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-89k2s" (UniqueName: "kubernetes.io/projected/f4602ac2-652b-4026-9a0d-d72162bb3258-kube-api-access-89k2s") pod "network-check-target-bbhx5" (UID: "f4602ac2-652b-4026-9a0d-d72162bb3258") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:17:02.620501 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:02.620406 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xj8mh" Oct 30 01:17:02.620501 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:02.620448 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxz45" Oct 30 01:17:02.620729 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:02.620564 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xj8mh" podUID="94864d90-1dd8-472e-a414-29c6b6a4f8f6" Oct 30 01:17:02.620729 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:02.620615 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxz45" podUID="f10aab9c-d4e6-45b5-a703-146114a31329" Oct 30 01:17:02.874593 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:02.874506 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xfvmk" event={"ID":"a2ec1edc-b911-4537-ac00-f708ef6c9062","Type":"ContainerStarted","Data":"32be0af037ffcd51274d9969b94bf6804209ef4dcbb8e7058ca80b8086d1e9e7"} Oct 30 01:17:02.877125 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:02.877108 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2vsns_d332bbb4-c699-44e8-96ba-516d8ca03e95/ovn-acl-logging/0.log" Oct 30 01:17:02.877475 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:02.877458 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" event={"ID":"d332bbb4-c699-44e8-96ba-516d8ca03e95","Type":"ContainerStarted","Data":"2f781b6cbdba9a4b7285549b5ffa6f4a28fa82e99fb3515552e211f4288aceff"} Oct 30 01:17:02.877526 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:02.877516 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" event={"ID":"d332bbb4-c699-44e8-96ba-516d8ca03e95","Type":"ContainerStarted","Data":"d83d7f1e2f0d70b42207e1b3db3388e4469fd2d1dd888ce5829a2ae1f1f10e6f"} Oct 30 01:17:02.877571 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:02.877562 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" event={"ID":"d332bbb4-c699-44e8-96ba-516d8ca03e95","Type":"ContainerStarted","Data":"0fc1cc18e7f1a48f36783c533c9181d20c957c143883a064abc9569ec9714b0e"} Oct 30 01:17:02.948708 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:02.948653 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xfvmk" podStartSLOduration=7.554436352 podStartE2EDuration="17.948638481s" totalImagesPullingTime="10.394202129s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:16:45 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:02.948322821 +0000 UTC m=+17.844300804" watchObservedRunningTime="2026-10-30 01:17:02.948638481 +0000 UTC m=+17.844616464" Oct 30 01:17:03.621037 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:03.621007 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbhx5" Oct 30 01:17:03.621179 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:03.621109 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbhx5" podUID="f4602ac2-652b-4026-9a0d-d72162bb3258" Oct 30 01:17:04.620680 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:04.620638 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xj8mh" Oct 30 01:17:04.620680 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:04.620670 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxz45" Oct 30 01:17:04.621120 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:04.620761 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxz45" podUID="f10aab9c-d4e6-45b5-a703-146114a31329" Oct 30 01:17:04.621120 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:04.620887 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xj8mh" podUID="94864d90-1dd8-472e-a414-29c6b6a4f8f6" Oct 30 01:17:04.885135 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:04.885065 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2vsns_d332bbb4-c699-44e8-96ba-516d8ca03e95/ovn-acl-logging/0.log" Oct 30 01:17:04.885416 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:04.885395 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" event={"ID":"d332bbb4-c699-44e8-96ba-516d8ca03e95","Type":"ContainerStarted","Data":"f174135424aeade539576b40a432bc8448054dc978a81a40c1fc5357640be670"} Oct 30 01:17:05.621387 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:05.621355 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbhx5" Oct 30 01:17:05.621824 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:05.621439 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbhx5" podUID="f4602ac2-652b-4026-9a0d-d72162bb3258" Oct 30 01:17:06.621064 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:06.621033 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xj8mh" Oct 30 01:17:06.621228 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:06.621047 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxz45" Oct 30 01:17:06.621228 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:06.621160 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xj8mh" podUID="94864d90-1dd8-472e-a414-29c6b6a4f8f6" Oct 30 01:17:06.621228 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:06.621198 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxz45" podUID="f10aab9c-d4e6-45b5-a703-146114a31329" Oct 30 01:17:06.892104 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:06.892037 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2vsns_d332bbb4-c699-44e8-96ba-516d8ca03e95/ovn-acl-logging/0.log" Oct 30 01:17:06.892667 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:06.892389 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" event={"ID":"d332bbb4-c699-44e8-96ba-516d8ca03e95","Type":"ContainerStarted","Data":"e89046dbbd1ce7a1cee349a7f12321a4246abe310cd0e620136c266b75598a92"} Oct 30 01:17:06.892725 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:06.892665 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:06.892772 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:06.892734 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:06.892850 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:06.892837 2573 scope.go:122] "RemoveContainer" containerID="c9e1e3083fc57e5d20d155273f7f436856af4d4f9acdf10b090e03b8c4a2a826" Oct 30 01:17:06.909977 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:06.909835 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:07.621376 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:07.621343 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbhx5" Oct 30 01:17:07.621572 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:07.621478 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbhx5" podUID="f4602ac2-652b-4026-9a0d-d72162bb3258" Oct 30 01:17:07.897818 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:07.897746 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2vsns_d332bbb4-c699-44e8-96ba-516d8ca03e95/ovn-acl-logging/0.log" Oct 30 01:17:07.898222 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:07.898118 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" event={"ID":"d332bbb4-c699-44e8-96ba-516d8ca03e95","Type":"ContainerStarted","Data":"4f245d0ed602be3d17970205977c85a2669f16f02d21de2b6f638f9d1bb083fd"} Oct 30 01:17:07.898377 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:07.898357 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:07.913595 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:07.913564 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:08.002341 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:08.002289 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" podStartSLOduration=8.281993628 podStartE2EDuration="23.002272629s" totalImagesPullingTime="14.720279001s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:08.001772896 +0000 UTC m=+22.897750893" watchObservedRunningTime="2026-10-30 01:17:08.002272629 +0000 UTC m=+22.898250612" Oct 30 01:17:08.375085 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:08.375038 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dxz45"] Oct 30 01:17:08.375260 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:08.375172 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxz45" Oct 30 01:17:08.375307 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:08.375291 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxz45" podUID="f10aab9c-d4e6-45b5-a703-146114a31329" Oct 30 01:17:08.377901 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:08.377880 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xj8mh"] Oct 30 01:17:08.378004 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:08.377972 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xj8mh" Oct 30 01:17:08.378076 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:08.378060 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xj8mh" podUID="94864d90-1dd8-472e-a414-29c6b6a4f8f6" Oct 30 01:17:08.378328 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:08.378311 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bbhx5"] Oct 30 01:17:08.378393 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:08.378383 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbhx5" Oct 30 01:17:08.378463 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:08.378448 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbhx5" podUID="f4602ac2-652b-4026-9a0d-d72162bb3258" Oct 30 01:17:09.623696 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:09.623671 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xj8mh" Oct 30 01:17:09.624185 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:09.623768 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xj8mh" podUID="94864d90-1dd8-472e-a414-29c6b6a4f8f6" Oct 30 01:17:10.620938 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:10.620845 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxz45" Oct 30 01:17:10.620938 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:10.620875 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbhx5" Oct 30 01:17:10.621140 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:10.620950 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxz45" podUID="f10aab9c-d4e6-45b5-a703-146114a31329" Oct 30 01:17:10.621140 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:10.621024 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbhx5" podUID="f4602ac2-652b-4026-9a0d-d72162bb3258" Oct 30 01:17:11.623930 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:11.623902 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xj8mh" Oct 30 01:17:11.624358 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:11.624027 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xj8mh" podUID="94864d90-1dd8-472e-a414-29c6b6a4f8f6" Oct 30 01:17:12.621211 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:12.621179 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbhx5" Oct 30 01:17:12.621372 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:12.621179 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxz45" Oct 30 01:17:12.621372 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:12.621285 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbhx5" podUID="f4602ac2-652b-4026-9a0d-d72162bb3258" Oct 30 01:17:12.621372 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:12.621354 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxz45" podUID="f10aab9c-d4e6-45b5-a703-146114a31329" Oct 30 01:17:13.621067 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:13.621037 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xj8mh" Oct 30 01:17:13.621442 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:13.621161 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xj8mh" podUID="94864d90-1dd8-472e-a414-29c6b6a4f8f6" Oct 30 01:17:13.742994 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:13.742966 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pf9jv_c819c74a-0f8d-4a5e-9e1b-830196f6b117/dns-node-resolver/0.log" Oct 30 01:17:14.620488 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:14.620446 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbhx5" Oct 30 01:17:14.620665 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:14.620582 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbhx5" podUID="f4602ac2-652b-4026-9a0d-d72162bb3258" Oct 30 01:17:14.620665 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:14.620627 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxz45" Oct 30 01:17:14.620791 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:14.620705 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxz45" podUID="f10aab9c-d4e6-45b5-a703-146114a31329" Oct 30 01:17:14.690205 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:14.690179 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lm86p_ecb9d3db-582f-41a0-945d-7bd7e4892ff8/node-ca/0.log" Oct 30 01:17:15.623912 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:15.623883 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xj8mh" Oct 30 01:17:15.624069 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:15.623975 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xj8mh" podUID="94864d90-1dd8-472e-a414-29c6b6a4f8f6" Oct 30 01:17:16.620542 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:16.620499 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbhx5" Oct 30 01:17:16.620949 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:16.620499 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxz45" Oct 30 01:17:16.620949 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:16.620622 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbhx5" podUID="f4602ac2-652b-4026-9a0d-d72162bb3258" Oct 30 01:17:16.620949 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:16.620665 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxz45" podUID="f10aab9c-d4e6-45b5-a703-146114a31329" Oct 30 01:17:17.623739 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:17.623711 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xj8mh" Oct 30 01:17:17.624206 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:17.623835 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xj8mh" podUID="94864d90-1dd8-472e-a414-29c6b6a4f8f6" Oct 30 01:17:18.354442 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:18.354375 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f10aab9c-d4e6-45b5-a703-146114a31329-original-pull-secret\") pod \"global-pull-secret-syncer-dxz45\" (UID: \"f10aab9c-d4e6-45b5-a703-146114a31329\") " pod="kube-system/global-pull-secret-syncer-dxz45" Oct 30 01:17:18.354442 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:18.354436 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94864d90-1dd8-472e-a414-29c6b6a4f8f6-metrics-certs\") pod \"network-metrics-daemon-xj8mh\" (UID: \"94864d90-1dd8-472e-a414-29c6b6a4f8f6\") " pod="openshift-multus/network-metrics-daemon-xj8mh" Oct 30 01:17:18.354725 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:18.354509 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 01:17:18.354725 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:18.354580 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f10aab9c-d4e6-45b5-a703-146114a31329-original-pull-secret podName:f10aab9c-d4e6-45b5-a703-146114a31329 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:50.354567004 +0000 UTC m=+65.250544965 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f10aab9c-d4e6-45b5-a703-146114a31329-original-pull-secret") pod "global-pull-secret-syncer-dxz45" (UID: "f10aab9c-d4e6-45b5-a703-146114a31329") : object "kube-system"/"original-pull-secret" not registered Oct 30 01:17:18.354725 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:18.354512 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:17:18.354725 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:18.354662 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94864d90-1dd8-472e-a414-29c6b6a4f8f6-metrics-certs podName:94864d90-1dd8-472e-a414-29c6b6a4f8f6 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:50.354641033 +0000 UTC m=+65.250618999 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94864d90-1dd8-472e-a414-29c6b6a4f8f6-metrics-certs") pod "network-metrics-daemon-xj8mh" (UID: "94864d90-1dd8-472e-a414-29c6b6a4f8f6") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 01:17:18.455609 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:18.455574 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89k2s\" (UniqueName: \"kubernetes.io/projected/f4602ac2-652b-4026-9a0d-d72162bb3258-kube-api-access-89k2s\") pod \"network-check-target-bbhx5\" (UID: \"f4602ac2-652b-4026-9a0d-d72162bb3258\") " pod="openshift-network-diagnostics/network-check-target-bbhx5" Oct 30 01:17:18.455766 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:18.455726 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 01:17:18.455766 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:18.455750 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 01:17:18.455766 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:18.455760 2573 projected.go:196] Error preparing data for projected volume kube-api-access-89k2s for pod openshift-network-diagnostics/network-check-target-bbhx5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:17:18.455892 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:18.455804 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f4602ac2-652b-4026-9a0d-d72162bb3258-kube-api-access-89k2s podName:f4602ac2-652b-4026-9a0d-d72162bb3258 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:50.455788972 +0000 UTC m=+65.351766933 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-89k2s" (UniqueName: "kubernetes.io/projected/f4602ac2-652b-4026-9a0d-d72162bb3258-kube-api-access-89k2s") pod "network-check-target-bbhx5" (UID: "f4602ac2-652b-4026-9a0d-d72162bb3258") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 01:17:18.621115 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:18.621040 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbhx5" Oct 30 01:17:18.621255 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:18.621040 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxz45" Oct 30 01:17:18.621255 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:18.621152 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbhx5" podUID="f4602ac2-652b-4026-9a0d-d72162bb3258" Oct 30 01:17:18.621255 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:18.621208 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxz45" podUID="f10aab9c-d4e6-45b5-a703-146114a31329" Oct 30 01:17:19.621385 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:19.621352 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xj8mh" Oct 30 01:17:19.621843 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:19.621505 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xj8mh" podUID="94864d90-1dd8-472e-a414-29c6b6a4f8f6" Oct 30 01:17:20.621079 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:20.621049 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxz45" Oct 30 01:17:20.621228 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:20.621157 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxz45" podUID="f10aab9c-d4e6-45b5-a703-146114a31329" Oct 30 01:17:20.621228 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:20.621210 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbhx5" Oct 30 01:17:20.621314 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:20.621297 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbhx5" podUID="f4602ac2-652b-4026-9a0d-d72162bb3258" Oct 30 01:17:21.621304 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:21.621269 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xj8mh" Oct 30 01:17:21.621794 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:21.621416 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xj8mh" podUID="94864d90-1dd8-472e-a414-29c6b6a4f8f6" Oct 30 01:17:22.620616 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:22.620585 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbhx5" Oct 30 01:17:22.620786 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:22.620624 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxz45" Oct 30 01:17:22.620786 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:22.620700 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbhx5" podUID="f4602ac2-652b-4026-9a0d-d72162bb3258" Oct 30 01:17:22.620864 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:22.620827 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxz45" podUID="f10aab9c-d4e6-45b5-a703-146114a31329" Oct 30 01:17:23.620581 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:23.620546 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xj8mh" Oct 30 01:17:23.621030 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:23.620687 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xj8mh" podUID="94864d90-1dd8-472e-a414-29c6b6a4f8f6" Oct 30 01:17:24.620572 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:24.620507 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbhx5" Oct 30 01:17:24.620758 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:24.620507 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxz45" Oct 30 01:17:24.620758 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:24.620640 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbhx5" podUID="f4602ac2-652b-4026-9a0d-d72162bb3258" Oct 30 01:17:24.620758 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:24.620741 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxz45" podUID="f10aab9c-d4e6-45b5-a703-146114a31329" Oct 30 01:17:25.623208 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:25.623176 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xj8mh" Oct 30 01:17:25.623618 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:25.623309 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xj8mh" podUID="94864d90-1dd8-472e-a414-29c6b6a4f8f6" Oct 30 01:17:26.620978 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:26.620897 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbhx5" Oct 30 01:17:26.620978 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:26.620932 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxz45" Oct 30 01:17:26.621149 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:26.621003 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbhx5" podUID="f4602ac2-652b-4026-9a0d-d72162bb3258" Oct 30 01:17:26.621149 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:26.621071 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxz45" podUID="f10aab9c-d4e6-45b5-a703-146114a31329" Oct 30 01:17:27.620841 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:27.620809 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xj8mh" Oct 30 01:17:27.621232 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:27.620918 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-xj8mh" podUID="94864d90-1dd8-472e-a414-29c6b6a4f8f6" Oct 30 01:17:28.620508 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:28.620480 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbhx5" Oct 30 01:17:28.620700 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:28.620649 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bbhx5" podUID="f4602ac2-652b-4026-9a0d-d72162bb3258" Oct 30 01:17:28.620700 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:28.620688 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxz45" Oct 30 01:17:28.620817 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:28.620755 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dxz45" podUID="f10aab9c-d4e6-45b5-a703-146114a31329" Oct 30 01:17:28.905964 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:28.905935 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-65.ec2.internal" event="NodeReady" Oct 30 01:17:28.906353 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:28.906050 2573 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 01:17:28.968024 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:28.967996 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hft6p"] Oct 30 01:17:28.969872 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:28.969856 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hft6p" Oct 30 01:17:28.975203 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:28.975176 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-cwzvk"] Oct 30 01:17:28.977014 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:28.976998 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cwzvk" Oct 30 01:17:28.987657 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:28.987629 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 01:17:28.987776 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:28.987700 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 01:17:28.987843 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:28.987825 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 01:17:28.987906 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:28.987894 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 01:17:28.989722 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:28.989704 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tk7cl"] Oct 30 01:17:28.991444 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:28.991420 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tk7cl" Oct 30 01:17:28.993773 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:28.993753 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hft6p"] Oct 30 01:17:28.998839 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:28.998646 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cwzvk"] Oct 30 01:17:29.016774 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.016747 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tk7cl"] Oct 30 01:17:29.087906 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.087879 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 01:17:29.088079 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.087919 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xmk59\"" Oct 30 01:17:29.088079 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.087930 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k4mfm\"" Oct 30 01:17:29.088153 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.088094 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 01:17:29.088153 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.088104 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ms58v\"" Oct 30 01:17:29.088376 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.088362 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 01:17:29.088652 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.088637 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 01:17:29.088692 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.088637 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 01:17:29.129758 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.129723 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd48f\" (UniqueName: \"kubernetes.io/projected/06f5cece-5273-44d1-90ba-67bf5dc0395b-kube-api-access-jd48f\") pod \"insights-runtime-extractor-hft6p\" (UID: \"06f5cece-5273-44d1-90ba-67bf5dc0395b\") " pod="openshift-insights/insights-runtime-extractor-hft6p" Oct 30 01:17:29.129920 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.129773 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4214a36a-fa6a-4290-aadc-2336881f84ba-config-volume\") pod \"dns-default-cwzvk\" (UID: \"4214a36a-fa6a-4290-aadc-2336881f84ba\") " pod="openshift-dns/dns-default-cwzvk" Oct 30 01:17:29.129920 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.129861 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4214a36a-fa6a-4290-aadc-2336881f84ba-tmp-dir\") pod \"dns-default-cwzvk\" (UID: \"4214a36a-fa6a-4290-aadc-2336881f84ba\") " pod="openshift-dns/dns-default-cwzvk" Oct 30 01:17:29.129920 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.129908 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/06f5cece-5273-44d1-90ba-67bf5dc0395b-crio-socket\") pod \"insights-runtime-extractor-hft6p\" (UID: \"06f5cece-5273-44d1-90ba-67bf5dc0395b\") " pod="openshift-insights/insights-runtime-extractor-hft6p" Oct 30 01:17:29.130013 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.129966 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4214a36a-fa6a-4290-aadc-2336881f84ba-metrics-tls\") pod \"dns-default-cwzvk\" (UID: \"4214a36a-fa6a-4290-aadc-2336881f84ba\") " pod="openshift-dns/dns-default-cwzvk" Oct 30 01:17:29.130013 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.129987 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/06f5cece-5273-44d1-90ba-67bf5dc0395b-data-volume\") pod \"insights-runtime-extractor-hft6p\" (UID: \"06f5cece-5273-44d1-90ba-67bf5dc0395b\") " pod="openshift-insights/insights-runtime-extractor-hft6p" Oct 30 01:17:29.130013 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.130009 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzhgj\" (UniqueName: \"kubernetes.io/projected/b981f97a-f238-4ef6-bbc8-e66cdd90dc90-kube-api-access-hzhgj\") pod \"ingress-canary-tk7cl\" (UID: \"b981f97a-f238-4ef6-bbc8-e66cdd90dc90\") " pod="openshift-ingress-canary/ingress-canary-tk7cl" Oct 30 01:17:29.130100 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.130027 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/06f5cece-5273-44d1-90ba-67bf5dc0395b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hft6p\" (UID: \"06f5cece-5273-44d1-90ba-67bf5dc0395b\") " pod="openshift-insights/insights-runtime-extractor-hft6p" Oct 30 01:17:29.130149 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.130129 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5k2j\" (UniqueName: \"kubernetes.io/projected/4214a36a-fa6a-4290-aadc-2336881f84ba-kube-api-access-t5k2j\") pod \"dns-default-cwzvk\" (UID: \"4214a36a-fa6a-4290-aadc-2336881f84ba\") " pod="openshift-dns/dns-default-cwzvk" Oct 30 01:17:29.130181 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.130158 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/06f5cece-5273-44d1-90ba-67bf5dc0395b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hft6p\" (UID: \"06f5cece-5273-44d1-90ba-67bf5dc0395b\") " pod="openshift-insights/insights-runtime-extractor-hft6p" Oct 30 01:17:29.130212 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.130187 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b981f97a-f238-4ef6-bbc8-e66cdd90dc90-cert\") pod \"ingress-canary-tk7cl\" (UID: \"b981f97a-f238-4ef6-bbc8-e66cdd90dc90\") " pod="openshift-ingress-canary/ingress-canary-tk7cl" Oct 30 01:17:29.230777 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.230698 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4214a36a-fa6a-4290-aadc-2336881f84ba-tmp-dir\") pod \"dns-default-cwzvk\" (UID: \"4214a36a-fa6a-4290-aadc-2336881f84ba\") " pod="openshift-dns/dns-default-cwzvk" Oct 30 01:17:29.230777 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.230759 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/06f5cece-5273-44d1-90ba-67bf5dc0395b-crio-socket\") pod \"insights-runtime-extractor-hft6p\" (UID: \"06f5cece-5273-44d1-90ba-67bf5dc0395b\") " pod="openshift-insights/insights-runtime-extractor-hft6p" Oct 30 01:17:29.230974 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.230786 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4214a36a-fa6a-4290-aadc-2336881f84ba-metrics-tls\") pod \"dns-default-cwzvk\" (UID: \"4214a36a-fa6a-4290-aadc-2336881f84ba\") " pod="openshift-dns/dns-default-cwzvk" Oct 30 01:17:29.230974 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.230808 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/06f5cece-5273-44d1-90ba-67bf5dc0395b-data-volume\") pod \"insights-runtime-extractor-hft6p\" (UID: \"06f5cece-5273-44d1-90ba-67bf5dc0395b\") " pod="openshift-insights/insights-runtime-extractor-hft6p" Oct 30 01:17:29.230974 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.230837 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzhgj\" (UniqueName: \"kubernetes.io/projected/b981f97a-f238-4ef6-bbc8-e66cdd90dc90-kube-api-access-hzhgj\") pod \"ingress-canary-tk7cl\" (UID: \"b981f97a-f238-4ef6-bbc8-e66cdd90dc90\") " pod="openshift-ingress-canary/ingress-canary-tk7cl" Oct 30 01:17:29.230974 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.230862 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/06f5cece-5273-44d1-90ba-67bf5dc0395b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hft6p\" (UID: \"06f5cece-5273-44d1-90ba-67bf5dc0395b\") " pod="openshift-insights/insights-runtime-extractor-hft6p" Oct 30 01:17:29.230974 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.230896 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5k2j\" (UniqueName: \"kubernetes.io/projected/4214a36a-fa6a-4290-aadc-2336881f84ba-kube-api-access-t5k2j\") pod \"dns-default-cwzvk\" (UID: \"4214a36a-fa6a-4290-aadc-2336881f84ba\") " pod="openshift-dns/dns-default-cwzvk" Oct 30 01:17:29.230974 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.230920 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/06f5cece-5273-44d1-90ba-67bf5dc0395b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hft6p\" (UID: \"06f5cece-5273-44d1-90ba-67bf5dc0395b\") " pod="openshift-insights/insights-runtime-extractor-hft6p" Oct 30 01:17:29.230974 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.230945 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b981f97a-f238-4ef6-bbc8-e66cdd90dc90-cert\") pod \"ingress-canary-tk7cl\" (UID: \"b981f97a-f238-4ef6-bbc8-e66cdd90dc90\") " pod="openshift-ingress-canary/ingress-canary-tk7cl" Oct 30 01:17:29.231293 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.230977 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd48f\" (UniqueName: \"kubernetes.io/projected/06f5cece-5273-44d1-90ba-67bf5dc0395b-kube-api-access-jd48f\") pod \"insights-runtime-extractor-hft6p\" (UID: \"06f5cece-5273-44d1-90ba-67bf5dc0395b\") " pod="openshift-insights/insights-runtime-extractor-hft6p" Oct 30 01:17:29.231293 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.231000 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/06f5cece-5273-44d1-90ba-67bf5dc0395b-crio-socket\") pod \"insights-runtime-extractor-hft6p\" (UID: \"06f5cece-5273-44d1-90ba-67bf5dc0395b\") " pod="openshift-insights/insights-runtime-extractor-hft6p" Oct 30 01:17:29.231293 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.231025 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4214a36a-fa6a-4290-aadc-2336881f84ba-config-volume\") pod \"dns-default-cwzvk\" (UID: \"4214a36a-fa6a-4290-aadc-2336881f84ba\") " pod="openshift-dns/dns-default-cwzvk" Oct 30 01:17:29.231293 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.231092 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4214a36a-fa6a-4290-aadc-2336881f84ba-tmp-dir\") pod \"dns-default-cwzvk\" (UID: \"4214a36a-fa6a-4290-aadc-2336881f84ba\") " pod="openshift-dns/dns-default-cwzvk" Oct 30 01:17:29.231293 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.231234 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/06f5cece-5273-44d1-90ba-67bf5dc0395b-data-volume\") pod \"insights-runtime-extractor-hft6p\" (UID: \"06f5cece-5273-44d1-90ba-67bf5dc0395b\") " pod="openshift-insights/insights-runtime-extractor-hft6p" Oct 30 01:17:29.231661 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.231637 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4214a36a-fa6a-4290-aadc-2336881f84ba-config-volume\") pod \"dns-default-cwzvk\" (UID: \"4214a36a-fa6a-4290-aadc-2336881f84ba\") " pod="openshift-dns/dns-default-cwzvk" Oct 30 01:17:29.231661 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.231664 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/06f5cece-5273-44d1-90ba-67bf5dc0395b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hft6p\" (UID: \"06f5cece-5273-44d1-90ba-67bf5dc0395b\") " pod="openshift-insights/insights-runtime-extractor-hft6p" Oct 30 01:17:29.234723 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.234700 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/06f5cece-5273-44d1-90ba-67bf5dc0395b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hft6p\" (UID: \"06f5cece-5273-44d1-90ba-67bf5dc0395b\") " pod="openshift-insights/insights-runtime-extractor-hft6p" Oct 30 01:17:29.234832 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.234817 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4214a36a-fa6a-4290-aadc-2336881f84ba-metrics-tls\") pod \"dns-default-cwzvk\" (UID: \"4214a36a-fa6a-4290-aadc-2336881f84ba\") " pod="openshift-dns/dns-default-cwzvk" Oct 30 01:17:29.235098 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.235079 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b981f97a-f238-4ef6-bbc8-e66cdd90dc90-cert\") pod \"ingress-canary-tk7cl\" (UID: \"b981f97a-f238-4ef6-bbc8-e66cdd90dc90\") " pod="openshift-ingress-canary/ingress-canary-tk7cl" Oct 30 01:17:29.255641 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.255616 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5k2j\" (UniqueName: \"kubernetes.io/projected/4214a36a-fa6a-4290-aadc-2336881f84ba-kube-api-access-t5k2j\") pod \"dns-default-cwzvk\" (UID: \"4214a36a-fa6a-4290-aadc-2336881f84ba\") " pod="openshift-dns/dns-default-cwzvk" Oct 30 01:17:29.255641 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.255631 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzhgj\" (UniqueName: \"kubernetes.io/projected/b981f97a-f238-4ef6-bbc8-e66cdd90dc90-kube-api-access-hzhgj\") pod \"ingress-canary-tk7cl\" (UID: \"b981f97a-f238-4ef6-bbc8-e66cdd90dc90\") " pod="openshift-ingress-canary/ingress-canary-tk7cl" Oct 30 01:17:29.255780 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.255635 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd48f\" (UniqueName: \"kubernetes.io/projected/06f5cece-5273-44d1-90ba-67bf5dc0395b-kube-api-access-jd48f\") pod \"insights-runtime-extractor-hft6p\" (UID: \"06f5cece-5273-44d1-90ba-67bf5dc0395b\") " pod="openshift-insights/insights-runtime-extractor-hft6p" Oct 30 01:17:29.281544 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.281510 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hft6p" Oct 30 01:17:29.288061 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.288043 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cwzvk" Oct 30 01:17:29.303000 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.302973 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tk7cl" Oct 30 01:17:29.602142 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.602110 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hft6p"] Oct 30 01:17:29.605827 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.605798 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cwzvk"] Oct 30 01:17:29.606167 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:17:29.606135 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06f5cece_5273_44d1_90ba_67bf5dc0395b.slice/crio-70df9bdcacf94f607d6ba53601e4133c206acac378e1e5de5ca27634d718bc16 WatchSource:0}: Error finding container 70df9bdcacf94f607d6ba53601e4133c206acac378e1e5de5ca27634d718bc16: Status 404 returned error can't find the container with id 70df9bdcacf94f607d6ba53601e4133c206acac378e1e5de5ca27634d718bc16 Oct 30 01:17:29.610420 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:17:29.610404 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4214a36a_fa6a_4290_aadc_2336881f84ba.slice/crio-59c2774dd37f74558ea7652843c6b6f576330ff6c1df4affcb6e60d147d5aa10 WatchSource:0}: Error finding container 59c2774dd37f74558ea7652843c6b6f576330ff6c1df4affcb6e60d147d5aa10: Status 404 returned error can't find the container with id 59c2774dd37f74558ea7652843c6b6f576330ff6c1df4affcb6e60d147d5aa10 Oct 30 01:17:29.617125 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.617099 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tk7cl"] Oct 30 01:17:29.620472 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.620452 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xj8mh" Oct 30 01:17:29.620824 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:17:29.620654 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb981f97a_f238_4ef6_bbc8_e66cdd90dc90.slice/crio-d56876b5d0a50691f4c01b9e84fa5f3de12509e574beb636d58a060e7cc7fe9f WatchSource:0}: Error finding container d56876b5d0a50691f4c01b9e84fa5f3de12509e574beb636d58a060e7cc7fe9f: Status 404 returned error can't find the container with id d56876b5d0a50691f4c01b9e84fa5f3de12509e574beb636d58a060e7cc7fe9f Oct 30 01:17:29.635917 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.635893 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 01:17:29.636061 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.636044 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xp5gx\"" Oct 30 01:17:29.948553 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.948453 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cwzvk" event={"ID":"4214a36a-fa6a-4290-aadc-2336881f84ba","Type":"ContainerStarted","Data":"59c2774dd37f74558ea7652843c6b6f576330ff6c1df4affcb6e60d147d5aa10"} Oct 30 01:17:29.949949 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.949922 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hft6p" event={"ID":"06f5cece-5273-44d1-90ba-67bf5dc0395b","Type":"ContainerStarted","Data":"3e85e0c5b479f5259cfeb686c335a89e194548c2b17c8e99df1de6b436041456"} Oct 30 01:17:29.950080 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.949968 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hft6p" event={"ID":"06f5cece-5273-44d1-90ba-67bf5dc0395b","Type":"ContainerStarted","Data":"70df9bdcacf94f607d6ba53601e4133c206acac378e1e5de5ca27634d718bc16"} Oct 30 01:17:29.951054 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:29.951032 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tk7cl" event={"ID":"b981f97a-f238-4ef6-bbc8-e66cdd90dc90","Type":"ContainerStarted","Data":"d56876b5d0a50691f4c01b9e84fa5f3de12509e574beb636d58a060e7cc7fe9f"} Oct 30 01:17:30.620972 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:30.620890 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxz45" Oct 30 01:17:30.620972 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:30.620937 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbhx5" Oct 30 01:17:30.637910 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:30.637769 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 01:17:30.637910 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:30.637883 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 01:17:30.651158 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:30.651126 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-wjscp\"" Oct 30 01:17:30.651266 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:30.651206 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 01:17:30.651266 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:30.651257 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-n79rv\"" Oct 30 01:17:30.685629 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:30.685599 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-9bf55565f-w7bkn"] Oct 30 01:17:30.687773 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:30.687751 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9bf55565f-w7bkn" Oct 30 01:17:30.717934 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:30.717900 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9bf55565f-w7bkn"] Oct 30 01:17:30.727579 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:30.727553 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 01:17:30.727712 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:30.727617 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 01:17:30.727712 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:30.727627 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 01:17:30.727712 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:30.727553 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 01:17:30.727712 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:30.727675 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 01:17:30.739313 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:30.739286 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 01:17:30.739575 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:30.739554 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 01:17:30.739658 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:30.739558 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-nbfct\"" Oct 30 01:17:30.845822 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:30.845787 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8e7e744e-9806-48ca-9d02-94bb07752f36-oauth-serving-cert\") pod \"console-9bf55565f-w7bkn\" (UID: \"8e7e744e-9806-48ca-9d02-94bb07752f36\") " pod="openshift-console/console-9bf55565f-w7bkn" Oct 30 01:17:30.845978 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:30.845850 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e7e744e-9806-48ca-9d02-94bb07752f36-console-serving-cert\") pod \"console-9bf55565f-w7bkn\" (UID: \"8e7e744e-9806-48ca-9d02-94bb07752f36\") " pod="openshift-console/console-9bf55565f-w7bkn" Oct 30 01:17:30.845978 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:30.845866 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8e7e744e-9806-48ca-9d02-94bb07752f36-console-oauth-config\") pod \"console-9bf55565f-w7bkn\" (UID: \"8e7e744e-9806-48ca-9d02-94bb07752f36\") " pod="openshift-console/console-9bf55565f-w7bkn" Oct 30 01:17:30.845978 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:30.845882 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz5xc\" (UniqueName: \"kubernetes.io/projected/8e7e744e-9806-48ca-9d02-94bb07752f36-kube-api-access-zz5xc\") pod \"console-9bf55565f-w7bkn\" (UID: \"8e7e744e-9806-48ca-9d02-94bb07752f36\") " pod="openshift-console/console-9bf55565f-w7bkn" Oct 30 01:17:30.845978 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:30.845928 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8e7e744e-9806-48ca-9d02-94bb07752f36-service-ca\") pod \"console-9bf55565f-w7bkn\" (UID: \"8e7e744e-9806-48ca-9d02-94bb07752f36\") " pod="openshift-console/console-9bf55565f-w7bkn" Oct 30 01:17:30.845978 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:30.845977 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e7e744e-9806-48ca-9d02-94bb07752f36-tmp\") pod \"console-9bf55565f-w7bkn\" (UID: \"8e7e744e-9806-48ca-9d02-94bb07752f36\") " pod="openshift-console/console-9bf55565f-w7bkn" Oct 30 01:17:30.846279 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:30.845996 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8e7e744e-9806-48ca-9d02-94bb07752f36-console-config\") pod \"console-9bf55565f-w7bkn\" (UID: \"8e7e744e-9806-48ca-9d02-94bb07752f36\") " pod="openshift-console/console-9bf55565f-w7bkn" Oct 30 01:17:30.946377 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:30.946289 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8e7e744e-9806-48ca-9d02-94bb07752f36-oauth-serving-cert\") pod \"console-9bf55565f-w7bkn\" (UID: \"8e7e744e-9806-48ca-9d02-94bb07752f36\") " pod="openshift-console/console-9bf55565f-w7bkn" Oct 30 01:17:30.946377 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:30.946359 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e7e744e-9806-48ca-9d02-94bb07752f36-console-serving-cert\") pod \"console-9bf55565f-w7bkn\" (UID: \"8e7e744e-9806-48ca-9d02-94bb07752f36\") " pod="openshift-console/console-9bf55565f-w7bkn" Oct 30 01:17:30.947310 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:30.946383 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8e7e744e-9806-48ca-9d02-94bb07752f36-console-oauth-config\") pod \"console-9bf55565f-w7bkn\" (UID: \"8e7e744e-9806-48ca-9d02-94bb07752f36\") " pod="openshift-console/console-9bf55565f-w7bkn" Oct 30 01:17:30.947310 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:30.946406 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz5xc\" (UniqueName: \"kubernetes.io/projected/8e7e744e-9806-48ca-9d02-94bb07752f36-kube-api-access-zz5xc\") pod \"console-9bf55565f-w7bkn\" (UID: \"8e7e744e-9806-48ca-9d02-94bb07752f36\") " pod="openshift-console/console-9bf55565f-w7bkn" Oct 30 01:17:30.947310 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:30.946454 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8e7e744e-9806-48ca-9d02-94bb07752f36-service-ca\") pod \"console-9bf55565f-w7bkn\" (UID: \"8e7e744e-9806-48ca-9d02-94bb07752f36\") " pod="openshift-console/console-9bf55565f-w7bkn" Oct 30 01:17:30.947310 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:30.946509 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e7e744e-9806-48ca-9d02-94bb07752f36-tmp\") pod \"console-9bf55565f-w7bkn\" (UID: \"8e7e744e-9806-48ca-9d02-94bb07752f36\") " pod="openshift-console/console-9bf55565f-w7bkn" Oct 30 01:17:30.947310 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:30.946559 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8e7e744e-9806-48ca-9d02-94bb07752f36-console-config\") pod \"console-9bf55565f-w7bkn\" (UID: \"8e7e744e-9806-48ca-9d02-94bb07752f36\") " pod="openshift-console/console-9bf55565f-w7bkn" Oct 30 01:17:30.947310 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:30.947121 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8e7e744e-9806-48ca-9d02-94bb07752f36-oauth-serving-cert\") pod \"console-9bf55565f-w7bkn\" (UID: \"8e7e744e-9806-48ca-9d02-94bb07752f36\") " pod="openshift-console/console-9bf55565f-w7bkn" Oct 30 01:17:30.947310 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:30.947158 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8e7e744e-9806-48ca-9d02-94bb07752f36-service-ca\") pod \"console-9bf55565f-w7bkn\" (UID: \"8e7e744e-9806-48ca-9d02-94bb07752f36\") " pod="openshift-console/console-9bf55565f-w7bkn" Oct 30 01:17:30.947659 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:30.947322 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e7e744e-9806-48ca-9d02-94bb07752f36-tmp\") pod \"console-9bf55565f-w7bkn\" (UID: \"8e7e744e-9806-48ca-9d02-94bb07752f36\") " pod="openshift-console/console-9bf55565f-w7bkn" Oct 30 01:17:30.947772 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:30.947752 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8e7e744e-9806-48ca-9d02-94bb07752f36-console-config\") pod \"console-9bf55565f-w7bkn\" (UID: \"8e7e744e-9806-48ca-9d02-94bb07752f36\") " pod="openshift-console/console-9bf55565f-w7bkn" Oct 30 01:17:30.950024 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:30.949997 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e7e744e-9806-48ca-9d02-94bb07752f36-console-serving-cert\") pod \"console-9bf55565f-w7bkn\" (UID: \"8e7e744e-9806-48ca-9d02-94bb07752f36\") " pod="openshift-console/console-9bf55565f-w7bkn" Oct 30 01:17:30.951205 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:30.951185 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8e7e744e-9806-48ca-9d02-94bb07752f36-console-oauth-config\") pod \"console-9bf55565f-w7bkn\" (UID: \"8e7e744e-9806-48ca-9d02-94bb07752f36\") " pod="openshift-console/console-9bf55565f-w7bkn" Oct 30 01:17:30.957084 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:30.957055 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hft6p" event={"ID":"06f5cece-5273-44d1-90ba-67bf5dc0395b","Type":"ContainerStarted","Data":"7f386037ecf20f02825606391976150200c8648ec9f3dfa709ed786156a18e94"} Oct 30 01:17:30.968058 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:30.968038 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz5xc\" (UniqueName: \"kubernetes.io/projected/8e7e744e-9806-48ca-9d02-94bb07752f36-kube-api-access-zz5xc\") pod \"console-9bf55565f-w7bkn\" (UID: \"8e7e744e-9806-48ca-9d02-94bb07752f36\") " pod="openshift-console/console-9bf55565f-w7bkn" Oct 30 01:17:30.999449 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:30.999412 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9bf55565f-w7bkn" Oct 30 01:17:31.306591 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:31.306477 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9bf55565f-w7bkn"] Oct 30 01:17:31.312018 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:17:31.311548 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e7e744e_9806_48ca_9d02_94bb07752f36.slice/crio-041c582efbcd1445b58cc13ef4091c174b65eb6004559fe5429f4cacc4235d4f WatchSource:0}: Error finding container 041c582efbcd1445b58cc13ef4091c174b65eb6004559fe5429f4cacc4235d4f: Status 404 returned error can't find the container with id 041c582efbcd1445b58cc13ef4091c174b65eb6004559fe5429f4cacc4235d4f Oct 30 01:17:31.606286 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:31.606261 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pf9jv_c819c74a-0f8d-4a5e-9e1b-830196f6b117/dns-node-resolver/0.log" Oct 30 01:17:31.962207 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:31.962168 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hft6p" event={"ID":"06f5cece-5273-44d1-90ba-67bf5dc0395b","Type":"ContainerStarted","Data":"93d54af1932bd825300edf338ca625f50579ae6e2d8abe0ed757d0dae1d5b40d"} Oct 30 01:17:31.964665 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:31.964633 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cwzvk" event={"ID":"4214a36a-fa6a-4290-aadc-2336881f84ba","Type":"ContainerStarted","Data":"e3dd4715079ebdcd022ff3820b47bf04a95c46474260b8dda7c5034c36c0d1e2"} Oct 30 01:17:31.964793 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:31.964672 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cwzvk" event={"ID":"4214a36a-fa6a-4290-aadc-2336881f84ba","Type":"ContainerStarted","Data":"f992f4f7a322b20751b80e24c6080c76c00cae1c25620a64d298537ed0321c23"} Oct 30 01:17:31.965106 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:31.965085 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-cwzvk" Oct 30 01:17:31.966424 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:31.966404 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9bf55565f-w7bkn" event={"ID":"8e7e744e-9806-48ca-9d02-94bb07752f36","Type":"ContainerStarted","Data":"041c582efbcd1445b58cc13ef4091c174b65eb6004559fe5429f4cacc4235d4f"} Oct 30 01:17:32.005797 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:32.005740 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hft6p" podStartSLOduration=1.885297924 podStartE2EDuration="4.005726435s" totalImagesPullingTime="2.120428511s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 01:17:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:32.004464866 +0000 UTC m=+46.900442849" watchObservedRunningTime="2026-10-30 01:17:32.005726435 +0000 UTC m=+46.901704411" Oct 30 01:17:32.047357 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:32.047285 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-cwzvk" podStartSLOduration=2.255792882 podStartE2EDuration="4.047270556s" totalImagesPullingTime="1.791477674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:32.046412774 +0000 UTC m=+46.942390760" watchObservedRunningTime="2026-10-30 01:17:32.047270556 +0000 UTC m=+46.943248538" Oct 30 01:17:32.806304 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:32.806270 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lm86p_ecb9d3db-582f-41a0-945d-7bd7e4892ff8/node-ca/0.log" Oct 30 01:17:32.896067 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:32.896032 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-8qcgr"] Oct 30 01:17:32.898179 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:32.898159 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-8qcgr" Oct 30 01:17:32.942132 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:32.940769 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 01:17:32.942132 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:32.940867 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-gpws7\"" Oct 30 01:17:32.942132 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:32.940924 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 01:17:32.942132 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:32.940967 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 01:17:32.942132 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:32.942073 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-8qcgr"] Oct 30 01:17:32.953881 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:32.953855 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 01:17:32.954009 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:32.953897 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 01:17:32.964679 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:32.964649 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9b098bdf-b302-4954-907b-eadab89054a0-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-8qcgr\" (UID: \"9b098bdf-b302-4954-907b-eadab89054a0\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8qcgr" Oct 30 01:17:32.965176 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:32.964818 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b098bdf-b302-4954-907b-eadab89054a0-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-8qcgr\" (UID: \"9b098bdf-b302-4954-907b-eadab89054a0\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8qcgr" Oct 30 01:17:32.965176 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:32.964888 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nb6l\" (UniqueName: \"kubernetes.io/projected/9b098bdf-b302-4954-907b-eadab89054a0-kube-api-access-5nb6l\") pod \"prometheus-operator-9cd89b74c-8qcgr\" (UID: \"9b098bdf-b302-4954-907b-eadab89054a0\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8qcgr" Oct 30 01:17:32.965176 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:32.964968 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9b098bdf-b302-4954-907b-eadab89054a0-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-8qcgr\" (UID: \"9b098bdf-b302-4954-907b-eadab89054a0\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8qcgr" Oct 30 01:17:32.970946 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:32.970918 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tk7cl" event={"ID":"b981f97a-f238-4ef6-bbc8-e66cdd90dc90","Type":"ContainerStarted","Data":"e14942c03240b3da3774bad4bcd12b7bdda3ce547ba2b23e89d71651c7ef4499"} Oct 30 01:17:33.014040 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:33.013983 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-tk7cl" podStartSLOduration=2.212498937 podStartE2EDuration="5.013968107s" totalImagesPullingTime="2.80146917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:33.013476475 +0000 UTC m=+47.909454459" watchObservedRunningTime="2026-10-30 01:17:33.013968107 +0000 UTC m=+47.909946090" Oct 30 01:17:33.065932 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:33.065858 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b098bdf-b302-4954-907b-eadab89054a0-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-8qcgr\" (UID: \"9b098bdf-b302-4954-907b-eadab89054a0\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8qcgr" Oct 30 01:17:33.065932 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:33.065901 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nb6l\" (UniqueName: \"kubernetes.io/projected/9b098bdf-b302-4954-907b-eadab89054a0-kube-api-access-5nb6l\") pod \"prometheus-operator-9cd89b74c-8qcgr\" (UID: \"9b098bdf-b302-4954-907b-eadab89054a0\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8qcgr" Oct 30 01:17:33.066141 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:33.065963 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9b098bdf-b302-4954-907b-eadab89054a0-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-8qcgr\" (UID: \"9b098bdf-b302-4954-907b-eadab89054a0\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8qcgr" Oct 30 01:17:33.066141 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:33.065994 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9b098bdf-b302-4954-907b-eadab89054a0-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-8qcgr\" (UID: \"9b098bdf-b302-4954-907b-eadab89054a0\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8qcgr" Oct 30 01:17:33.066552 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:33.066516 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b098bdf-b302-4954-907b-eadab89054a0-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-8qcgr\" (UID: \"9b098bdf-b302-4954-907b-eadab89054a0\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8qcgr" Oct 30 01:17:33.069023 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:33.068995 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9b098bdf-b302-4954-907b-eadab89054a0-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-8qcgr\" (UID: \"9b098bdf-b302-4954-907b-eadab89054a0\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8qcgr" Oct 30 01:17:33.069118 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:33.069067 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9b098bdf-b302-4954-907b-eadab89054a0-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-8qcgr\" (UID: \"9b098bdf-b302-4954-907b-eadab89054a0\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8qcgr" Oct 30 01:17:33.086256 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:33.086234 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nb6l\" (UniqueName: \"kubernetes.io/projected/9b098bdf-b302-4954-907b-eadab89054a0-kube-api-access-5nb6l\") pod \"prometheus-operator-9cd89b74c-8qcgr\" (UID: \"9b098bdf-b302-4954-907b-eadab89054a0\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8qcgr" Oct 30 01:17:33.209976 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:33.209942 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-8qcgr" Oct 30 01:17:33.509210 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:33.509131 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-8qcgr"] Oct 30 01:17:33.513089 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:17:33.513062 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b098bdf_b302_4954_907b_eadab89054a0.slice/crio-47d43ed550365a099b9f7367bde105cb6ea4ca15b5aa598ca8465137f2650ea4 WatchSource:0}: Error finding container 47d43ed550365a099b9f7367bde105cb6ea4ca15b5aa598ca8465137f2650ea4: Status 404 returned error can't find the container with id 47d43ed550365a099b9f7367bde105cb6ea4ca15b5aa598ca8465137f2650ea4 Oct 30 01:17:33.606635 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:33.606603 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tk7cl_b981f97a-f238-4ef6-bbc8-e66cdd90dc90/serve-healthcheck-canary/0.log" Oct 30 01:17:33.975887 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:33.975805 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-8qcgr" event={"ID":"9b098bdf-b302-4954-907b-eadab89054a0","Type":"ContainerStarted","Data":"47d43ed550365a099b9f7367bde105cb6ea4ca15b5aa598ca8465137f2650ea4"} Oct 30 01:17:34.980960 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:34.980871 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9bf55565f-w7bkn" event={"ID":"8e7e744e-9806-48ca-9d02-94bb07752f36","Type":"ContainerStarted","Data":"fc1bd0a7e19c1ff001def32af6a08009019658216ffa26169a4abeb435cdfeb0"} Oct 30 01:17:34.982679 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:34.982658 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-8qcgr" event={"ID":"9b098bdf-b302-4954-907b-eadab89054a0","Type":"ContainerStarted","Data":"fc0072a44ae69407e066a75dd5f9500a93a7a9d3e7696f73c3f3fdc97960f183"} Oct 30 01:17:34.982790 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:34.982694 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-8qcgr" event={"ID":"9b098bdf-b302-4954-907b-eadab89054a0","Type":"ContainerStarted","Data":"4f0ea7e052fcbaf84781db19b23da11665d5ebca9fd92d10523f3b7bb5fac78e"} Oct 30 01:17:35.067851 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:35.067806 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-9bf55565f-w7bkn" podStartSLOduration=2.221206931 podStartE2EDuration="5.067792787s" totalImagesPullingTime="2.846585856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:35.022624665 +0000 UTC m=+49.918602648" watchObservedRunningTime="2026-10-30 01:17:35.067792787 +0000 UTC m=+49.963770770" Oct 30 01:17:35.068014 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:35.067904 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-8qcgr" podStartSLOduration=1.923413535 podStartE2EDuration="3.067899807s" totalImagesPullingTime="1.144486272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:35.067199308 +0000 UTC m=+49.963177290" watchObservedRunningTime="2026-10-30 01:17:35.067899807 +0000 UTC m=+49.963877804" Oct 30 01:17:37.561405 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:37.561368 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-tdj6l"] Oct 30 01:17:37.563592 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:37.563576 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:37.583379 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:37.583356 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 01:17:37.600006 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:37.599977 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/806dfce0-6207-4feb-bee4-ce73cfd4b552-node-exporter-wtmp\") pod \"node-exporter-tdj6l\" (UID: \"806dfce0-6207-4feb-bee4-ce73cfd4b552\") " pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:37.600114 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:37.600021 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/806dfce0-6207-4feb-bee4-ce73cfd4b552-sys\") pod \"node-exporter-tdj6l\" (UID: \"806dfce0-6207-4feb-bee4-ce73cfd4b552\") " pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:37.600159 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:37.600126 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/806dfce0-6207-4feb-bee4-ce73cfd4b552-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tdj6l\" (UID: \"806dfce0-6207-4feb-bee4-ce73cfd4b552\") " pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:37.600159 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:37.600155 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgt8r\" (UniqueName: \"kubernetes.io/projected/806dfce0-6207-4feb-bee4-ce73cfd4b552-kube-api-access-bgt8r\") pod \"node-exporter-tdj6l\" (UID: \"806dfce0-6207-4feb-bee4-ce73cfd4b552\") " pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:37.600234 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:37.600174 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/806dfce0-6207-4feb-bee4-ce73cfd4b552-root\") pod \"node-exporter-tdj6l\" (UID: \"806dfce0-6207-4feb-bee4-ce73cfd4b552\") " pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:37.600234 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:37.600189 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/806dfce0-6207-4feb-bee4-ce73cfd4b552-node-exporter-tls\") pod \"node-exporter-tdj6l\" (UID: \"806dfce0-6207-4feb-bee4-ce73cfd4b552\") " pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:37.600323 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:37.600289 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/806dfce0-6207-4feb-bee4-ce73cfd4b552-node-exporter-textfile\") pod \"node-exporter-tdj6l\" (UID: \"806dfce0-6207-4feb-bee4-ce73cfd4b552\") " pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:37.600381 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:37.600334 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/806dfce0-6207-4feb-bee4-ce73cfd4b552-node-exporter-accelerators-collector-config\") pod \"node-exporter-tdj6l\" (UID: \"806dfce0-6207-4feb-bee4-ce73cfd4b552\") " pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:37.600381 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:37.600366 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/806dfce0-6207-4feb-bee4-ce73cfd4b552-metrics-client-ca\") pod \"node-exporter-tdj6l\" (UID: \"806dfce0-6207-4feb-bee4-ce73cfd4b552\") " pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:37.608448 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:37.608428 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 01:17:37.608665 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:37.608649 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vthnr\"" Oct 30 01:17:37.608712 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:37.608676 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 01:17:37.701177 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:37.701138 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgt8r\" (UniqueName: \"kubernetes.io/projected/806dfce0-6207-4feb-bee4-ce73cfd4b552-kube-api-access-bgt8r\") pod \"node-exporter-tdj6l\" (UID: \"806dfce0-6207-4feb-bee4-ce73cfd4b552\") " pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:37.701177 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:37.701178 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/806dfce0-6207-4feb-bee4-ce73cfd4b552-root\") pod \"node-exporter-tdj6l\" (UID: \"806dfce0-6207-4feb-bee4-ce73cfd4b552\") " pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:37.701381 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:37.701205 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/806dfce0-6207-4feb-bee4-ce73cfd4b552-node-exporter-tls\") pod \"node-exporter-tdj6l\" (UID: \"806dfce0-6207-4feb-bee4-ce73cfd4b552\") " pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:37.701381 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:37.701271 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/806dfce0-6207-4feb-bee4-ce73cfd4b552-root\") pod \"node-exporter-tdj6l\" (UID: \"806dfce0-6207-4feb-bee4-ce73cfd4b552\") " pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:37.701381 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:37.701326 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/806dfce0-6207-4feb-bee4-ce73cfd4b552-node-exporter-textfile\") pod \"node-exporter-tdj6l\" (UID: \"806dfce0-6207-4feb-bee4-ce73cfd4b552\") " pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:37.701478 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:37.701384 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/806dfce0-6207-4feb-bee4-ce73cfd4b552-node-exporter-accelerators-collector-config\") pod \"node-exporter-tdj6l\" (UID: \"806dfce0-6207-4feb-bee4-ce73cfd4b552\") " pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:37.701478 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:37.701436 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/806dfce0-6207-4feb-bee4-ce73cfd4b552-metrics-client-ca\") pod \"node-exporter-tdj6l\" (UID: \"806dfce0-6207-4feb-bee4-ce73cfd4b552\") " pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:37.701599 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:37.701578 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/806dfce0-6207-4feb-bee4-ce73cfd4b552-node-exporter-wtmp\") pod \"node-exporter-tdj6l\" (UID: \"806dfce0-6207-4feb-bee4-ce73cfd4b552\") " pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:37.701660 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:37.701637 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/806dfce0-6207-4feb-bee4-ce73cfd4b552-sys\") pod \"node-exporter-tdj6l\" (UID: \"806dfce0-6207-4feb-bee4-ce73cfd4b552\") " pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:37.701723 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:37.701709 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/806dfce0-6207-4feb-bee4-ce73cfd4b552-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tdj6l\" (UID: \"806dfce0-6207-4feb-bee4-ce73cfd4b552\") " pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:37.701775 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:37.701736 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/806dfce0-6207-4feb-bee4-ce73cfd4b552-node-exporter-wtmp\") pod \"node-exporter-tdj6l\" (UID: \"806dfce0-6207-4feb-bee4-ce73cfd4b552\") " pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:37.701775 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:37.701742 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/806dfce0-6207-4feb-bee4-ce73cfd4b552-node-exporter-textfile\") pod \"node-exporter-tdj6l\" (UID: \"806dfce0-6207-4feb-bee4-ce73cfd4b552\") " pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:37.701872 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:37.701740 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/806dfce0-6207-4feb-bee4-ce73cfd4b552-sys\") pod \"node-exporter-tdj6l\" (UID: \"806dfce0-6207-4feb-bee4-ce73cfd4b552\") " pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:37.701974 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:37.701953 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/806dfce0-6207-4feb-bee4-ce73cfd4b552-node-exporter-accelerators-collector-config\") pod \"node-exporter-tdj6l\" (UID: \"806dfce0-6207-4feb-bee4-ce73cfd4b552\") " pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:37.702033 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:37.701975 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/806dfce0-6207-4feb-bee4-ce73cfd4b552-metrics-client-ca\") pod \"node-exporter-tdj6l\" (UID: \"806dfce0-6207-4feb-bee4-ce73cfd4b552\") " pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:37.703567 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:37.703527 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/806dfce0-6207-4feb-bee4-ce73cfd4b552-node-exporter-tls\") pod \"node-exporter-tdj6l\" (UID: \"806dfce0-6207-4feb-bee4-ce73cfd4b552\") " pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:37.703733 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:37.703717 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/806dfce0-6207-4feb-bee4-ce73cfd4b552-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tdj6l\" (UID: \"806dfce0-6207-4feb-bee4-ce73cfd4b552\") " pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:37.726464 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:37.726436 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgt8r\" (UniqueName: \"kubernetes.io/projected/806dfce0-6207-4feb-bee4-ce73cfd4b552-kube-api-access-bgt8r\") pod \"node-exporter-tdj6l\" (UID: \"806dfce0-6207-4feb-bee4-ce73cfd4b552\") " pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:37.875290 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:37.875209 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:37.883890 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:17:37.883863 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod806dfce0_6207_4feb_bee4_ce73cfd4b552.slice/crio-b3c286849918fe011ae31026c2e3dcc3e0f10a61248956794ccb5750291206b1 WatchSource:0}: Error finding container b3c286849918fe011ae31026c2e3dcc3e0f10a61248956794ccb5750291206b1: Status 404 returned error can't find the container with id b3c286849918fe011ae31026c2e3dcc3e0f10a61248956794ccb5750291206b1 Oct 30 01:17:37.992902 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:37.992869 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tdj6l" event={"ID":"806dfce0-6207-4feb-bee4-ce73cfd4b552","Type":"ContainerStarted","Data":"b3c286849918fe011ae31026c2e3dcc3e0f10a61248956794ccb5750291206b1"} Oct 30 01:17:38.943700 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:38.943673 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 01:17:38.946034 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:38.946012 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:38.963191 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:38.963168 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 01:17:38.963191 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:38.963180 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 01:17:38.963387 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:38.963211 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 01:17:38.963387 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:38.963258 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 01:17:38.963682 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:38.963665 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 01:17:38.975835 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:38.975810 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-bvfsl\"" Oct 30 01:17:38.976348 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:38.976325 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 01:17:38.978367 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:38.978339 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 01:17:38.998410 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:38.998382 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tdj6l" event={"ID":"806dfce0-6207-4feb-bee4-ce73cfd4b552","Type":"ContainerStarted","Data":"2f21aa0a1d97280e413497f2f866bc7808fbc92d642ad487b104520c4005567a"} Oct 30 01:17:39.014196 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:39.014170 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d205185b-fc45-4a76-9719-20b00107abf1-config-out\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:39.014196 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:39.014198 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:39.014417 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:39.014221 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:39.014417 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:39.014270 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:39.014524 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:39.014414 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d205185b-fc45-4a76-9719-20b00107abf1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:39.014524 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:39.014448 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d205185b-fc45-4a76-9719-20b00107abf1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:39.014524 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:39.014516 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-config-volume\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:39.014713 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:39.014683 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:39.014762 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:39.014718 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-web-config\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:39.014813 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:39.014749 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d205185b-fc45-4a76-9719-20b00107abf1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:39.014813 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:39.014782 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:39.014907 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:39.014824 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d205185b-fc45-4a76-9719-20b00107abf1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:39.014907 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:39.014865 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcd88\" (UniqueName: \"kubernetes.io/projected/d205185b-fc45-4a76-9719-20b00107abf1-kube-api-access-mcd88\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:39.057246 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:39.057207 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 01:17:39.057384 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:39.057250 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 01:17:39.057384 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:39.057271 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 01:17:39.115373 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:39.115342 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-config-volume\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:39.115579 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:39.115391 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:39.115579 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:39.115408 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-web-config\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:39.115579 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:39.115428 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d205185b-fc45-4a76-9719-20b00107abf1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:39.115579 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:39.115563 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:39.115784 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:39.115595 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d205185b-fc45-4a76-9719-20b00107abf1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:39.115784 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:39.115634 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcd88\" (UniqueName: \"kubernetes.io/projected/d205185b-fc45-4a76-9719-20b00107abf1-kube-api-access-mcd88\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:39.115784 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:39.115687 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d205185b-fc45-4a76-9719-20b00107abf1-config-out\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:39.115784 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:39.115712 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:39.115784 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:39.115741 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:39.115784 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:39.115772 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:39.116079 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:39.115814 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d205185b-fc45-4a76-9719-20b00107abf1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:39.116079 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:39.115841 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d205185b-fc45-4a76-9719-20b00107abf1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:39.116079 ip-10-0-135-65 kubenswrapper[2573]: E1030 01:17:39.115988 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d205185b-fc45-4a76-9719-20b00107abf1-alertmanager-trusted-ca-bundle podName:d205185b-fc45-4a76-9719-20b00107abf1 nodeName:}" failed. No retries permitted until 2026-10-30 01:17:39.615970243 +0000 UTC m=+54.511948220 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/d205185b-fc45-4a76-9719-20b00107abf1-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "d205185b-fc45-4a76-9719-20b00107abf1") : configmap references non-existent config key: ca-bundle.crt Oct 30 01:17:39.116079 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:39.116018 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d205185b-fc45-4a76-9719-20b00107abf1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:39.116768 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:39.116716 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d205185b-fc45-4a76-9719-20b00107abf1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:39.118240 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:39.118218 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d205185b-fc45-4a76-9719-20b00107abf1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:39.119178 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:39.119028 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d205185b-fc45-4a76-9719-20b00107abf1-config-out\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:39.119267 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:39.119176 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:39.119267 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:39.119248 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:39.119267 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:39.119255 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:39.119552 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:39.119514 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:39.119631 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:39.119553 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-web-config\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:39.119684 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:39.119646 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-config-volume\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:39.120102 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:39.120081 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:39.135401 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:39.135353 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcd88\" (UniqueName: \"kubernetes.io/projected/d205185b-fc45-4a76-9719-20b00107abf1-kube-api-access-mcd88\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:39.620927 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:39.620883 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d205185b-fc45-4a76-9719-20b00107abf1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:39.621652 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:39.621631 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d205185b-fc45-4a76-9719-20b00107abf1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:39.862056 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:39.862025 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:39.923563 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:39.922760 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:40.003002 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:40.002971 2573 generic.go:356] "Generic (PLEG): container finished" podID="806dfce0-6207-4feb-bee4-ce73cfd4b552" containerID="2f21aa0a1d97280e413497f2f866bc7808fbc92d642ad487b104520c4005567a" exitCode=0 Oct 30 01:17:40.003406 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:40.003028 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tdj6l" event={"ID":"806dfce0-6207-4feb-bee4-ce73cfd4b552","Type":"ContainerDied","Data":"2f21aa0a1d97280e413497f2f866bc7808fbc92d642ad487b104520c4005567a"} Oct 30 01:17:40.217963 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:40.217938 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 01:17:40.219508 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:17:40.219484 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd205185b_fc45_4a76_9719_20b00107abf1.slice/crio-66cfed6041737b855c39214ebc841b78bd2197bc4a3cbd61dfec11a91da2bde2 WatchSource:0}: Error finding container 66cfed6041737b855c39214ebc841b78bd2197bc4a3cbd61dfec11a91da2bde2: Status 404 returned error can't find the container with id 66cfed6041737b855c39214ebc841b78bd2197bc4a3cbd61dfec11a91da2bde2 Oct 30 01:17:41.000593 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:41.000558 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9bf55565f-w7bkn" Oct 30 01:17:41.000755 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:41.000620 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9bf55565f-w7bkn" Oct 30 01:17:41.006111 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:41.006083 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9bf55565f-w7bkn" Oct 30 01:17:41.009704 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:41.009674 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tdj6l" event={"ID":"806dfce0-6207-4feb-bee4-ce73cfd4b552","Type":"ContainerStarted","Data":"4fae0984c2bb5d8f359945fc4c50d6b7d9db43d63e38c392682e24a989521984"} Oct 30 01:17:41.009848 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:41.009724 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-tdj6l" event={"ID":"806dfce0-6207-4feb-bee4-ce73cfd4b552","Type":"ContainerStarted","Data":"46dc2bedcc039974b14767904b11166876fa57fbd787ec294a5739fe672c2f62"} Oct 30 01:17:41.011080 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:41.011057 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d205185b-fc45-4a76-9719-20b00107abf1","Type":"ContainerStarted","Data":"66cfed6041737b855c39214ebc841b78bd2197bc4a3cbd61dfec11a91da2bde2"} Oct 30 01:17:41.015514 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:41.015492 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9bf55565f-w7bkn" Oct 30 01:17:41.096350 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:41.096295 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-tdj6l" podStartSLOduration=3.029043571 podStartE2EDuration="4.096282033s" totalImagesPullingTime="996.113997ms" totalInitContainerRuntime="71.124465ms" isStatefulPod=true podCreationTimestamp="2026-10-30 01:17:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:41.095602474 +0000 UTC m=+55.991580458" watchObservedRunningTime="2026-10-30 01:17:41.096282033 +0000 UTC m=+55.992260016" Oct 30 01:17:41.097736 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:41.097715 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b6fbb6f6f-9z4ws"] Oct 30 01:17:41.100139 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:41.100118 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b6fbb6f6f-9z4ws" Oct 30 01:17:41.119573 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:41.119543 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b6fbb6f6f-9z4ws"] Oct 30 01:17:41.131851 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:41.131815 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db13cab2-9b76-44b5-8577-9f8f0d3f292e-trusted-ca-bundle\") pod \"console-7b6fbb6f6f-9z4ws\" (UID: \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\") " pod="openshift-console/console-7b6fbb6f6f-9z4ws" Oct 30 01:17:41.131976 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:41.131871 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db13cab2-9b76-44b5-8577-9f8f0d3f292e-console-oauth-config\") pod \"console-7b6fbb6f6f-9z4ws\" (UID: \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\") " pod="openshift-console/console-7b6fbb6f6f-9z4ws" Oct 30 01:17:41.132051 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:41.132003 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db13cab2-9b76-44b5-8577-9f8f0d3f292e-console-config\") pod \"console-7b6fbb6f6f-9z4ws\" (UID: \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\") " pod="openshift-console/console-7b6fbb6f6f-9z4ws" Oct 30 01:17:41.132051 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:41.132039 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db13cab2-9b76-44b5-8577-9f8f0d3f292e-console-serving-cert\") pod \"console-7b6fbb6f6f-9z4ws\" (UID: \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\") " pod="openshift-console/console-7b6fbb6f6f-9z4ws" Oct 30 01:17:41.132154 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:41.132073 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db13cab2-9b76-44b5-8577-9f8f0d3f292e-tmp\") pod \"console-7b6fbb6f6f-9z4ws\" (UID: \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\") " pod="openshift-console/console-7b6fbb6f6f-9z4ws" Oct 30 01:17:41.132154 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:41.132118 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 01:17:41.132154 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:41.132136 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db13cab2-9b76-44b5-8577-9f8f0d3f292e-service-ca\") pod \"console-7b6fbb6f6f-9z4ws\" (UID: \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\") " pod="openshift-console/console-7b6fbb6f6f-9z4ws" Oct 30 01:17:41.132263 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:41.132198 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db13cab2-9b76-44b5-8577-9f8f0d3f292e-oauth-serving-cert\") pod \"console-7b6fbb6f6f-9z4ws\" (UID: \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\") " pod="openshift-console/console-7b6fbb6f6f-9z4ws" Oct 30 01:17:41.132263 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:41.132219 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7frp\" (UniqueName: \"kubernetes.io/projected/db13cab2-9b76-44b5-8577-9f8f0d3f292e-kube-api-access-l7frp\") pod \"console-7b6fbb6f6f-9z4ws\" (UID: \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\") " pod="openshift-console/console-7b6fbb6f6f-9z4ws" Oct 30 01:17:41.233235 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:41.233214 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db13cab2-9b76-44b5-8577-9f8f0d3f292e-tmp\") pod \"console-7b6fbb6f6f-9z4ws\" (UID: \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\") " pod="openshift-console/console-7b6fbb6f6f-9z4ws" Oct 30 01:17:41.233326 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:41.233247 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db13cab2-9b76-44b5-8577-9f8f0d3f292e-service-ca\") pod \"console-7b6fbb6f6f-9z4ws\" (UID: \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\") " pod="openshift-console/console-7b6fbb6f6f-9z4ws" Oct 30 01:17:41.233326 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:41.233288 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db13cab2-9b76-44b5-8577-9f8f0d3f292e-oauth-serving-cert\") pod \"console-7b6fbb6f6f-9z4ws\" (UID: \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\") " pod="openshift-console/console-7b6fbb6f6f-9z4ws" Oct 30 01:17:41.233326 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:41.233304 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7frp\" (UniqueName: \"kubernetes.io/projected/db13cab2-9b76-44b5-8577-9f8f0d3f292e-kube-api-access-l7frp\") pod \"console-7b6fbb6f6f-9z4ws\" (UID: \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\") " pod="openshift-console/console-7b6fbb6f6f-9z4ws" Oct 30 01:17:41.233459 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:41.233327 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db13cab2-9b76-44b5-8577-9f8f0d3f292e-trusted-ca-bundle\") pod \"console-7b6fbb6f6f-9z4ws\" (UID: \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\") " pod="openshift-console/console-7b6fbb6f6f-9z4ws" Oct 30 01:17:41.233459 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:41.233349 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db13cab2-9b76-44b5-8577-9f8f0d3f292e-console-oauth-config\") pod \"console-7b6fbb6f6f-9z4ws\" (UID: \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\") " pod="openshift-console/console-7b6fbb6f6f-9z4ws" Oct 30 01:17:41.233459 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:41.233402 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db13cab2-9b76-44b5-8577-9f8f0d3f292e-console-config\") pod \"console-7b6fbb6f6f-9z4ws\" (UID: \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\") " pod="openshift-console/console-7b6fbb6f6f-9z4ws" Oct 30 01:17:41.233459 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:41.233436 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db13cab2-9b76-44b5-8577-9f8f0d3f292e-console-serving-cert\") pod \"console-7b6fbb6f6f-9z4ws\" (UID: \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\") " pod="openshift-console/console-7b6fbb6f6f-9z4ws" Oct 30 01:17:41.234379 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:41.234080 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db13cab2-9b76-44b5-8577-9f8f0d3f292e-tmp\") pod \"console-7b6fbb6f6f-9z4ws\" (UID: \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\") " pod="openshift-console/console-7b6fbb6f6f-9z4ws" Oct 30 01:17:41.234379 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:41.234366 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db13cab2-9b76-44b5-8577-9f8f0d3f292e-oauth-serving-cert\") pod \"console-7b6fbb6f6f-9z4ws\" (UID: \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\") " pod="openshift-console/console-7b6fbb6f6f-9z4ws" Oct 30 01:17:41.234780 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:41.234573 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db13cab2-9b76-44b5-8577-9f8f0d3f292e-service-ca\") pod \"console-7b6fbb6f6f-9z4ws\" (UID: \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\") " pod="openshift-console/console-7b6fbb6f6f-9z4ws" Oct 30 01:17:41.234780 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:41.234655 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db13cab2-9b76-44b5-8577-9f8f0d3f292e-trusted-ca-bundle\") pod \"console-7b6fbb6f6f-9z4ws\" (UID: \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\") " pod="openshift-console/console-7b6fbb6f6f-9z4ws" Oct 30 01:17:41.234929 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:41.234827 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db13cab2-9b76-44b5-8577-9f8f0d3f292e-console-config\") pod \"console-7b6fbb6f6f-9z4ws\" (UID: \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\") " pod="openshift-console/console-7b6fbb6f6f-9z4ws" Oct 30 01:17:41.237345 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:41.236400 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db13cab2-9b76-44b5-8577-9f8f0d3f292e-console-oauth-config\") pod \"console-7b6fbb6f6f-9z4ws\" (UID: \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\") " pod="openshift-console/console-7b6fbb6f6f-9z4ws" Oct 30 01:17:41.239450 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:41.239432 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db13cab2-9b76-44b5-8577-9f8f0d3f292e-console-serving-cert\") pod \"console-7b6fbb6f6f-9z4ws\" (UID: \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\") " pod="openshift-console/console-7b6fbb6f6f-9z4ws" Oct 30 01:17:41.252907 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:41.252858 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7frp\" (UniqueName: \"kubernetes.io/projected/db13cab2-9b76-44b5-8577-9f8f0d3f292e-kube-api-access-l7frp\") pod \"console-7b6fbb6f6f-9z4ws\" (UID: \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\") " pod="openshift-console/console-7b6fbb6f6f-9z4ws" Oct 30 01:17:41.413892 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:41.413851 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b6fbb6f6f-9z4ws" Oct 30 01:17:41.706158 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:41.706127 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b6fbb6f6f-9z4ws"] Oct 30 01:17:41.709249 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:17:41.709217 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb13cab2_9b76_44b5_8577_9f8f0d3f292e.slice/crio-223f329baedfd1629bc7e494bc60ac6c4b654bcbc118acaff1ad96b96e5fc52a WatchSource:0}: Error finding container 223f329baedfd1629bc7e494bc60ac6c4b654bcbc118acaff1ad96b96e5fc52a: Status 404 returned error can't find the container with id 223f329baedfd1629bc7e494bc60ac6c4b654bcbc118acaff1ad96b96e5fc52a Oct 30 01:17:41.983362 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:41.983281 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-944876b99-29n9f"] Oct 30 01:17:41.985337 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:41.985317 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-944876b99-29n9f" Oct 30 01:17:42.002407 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:42.002377 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 01:17:42.002407 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:42.002388 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 01:17:42.002592 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:42.002412 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-54zkd\"" Oct 30 01:17:42.002592 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:42.002403 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 01:17:42.002592 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:42.002454 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 01:17:42.002592 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:42.002461 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-aeu5o81g4aq44\"" Oct 30 01:17:42.009347 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:42.009320 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-944876b99-29n9f"] Oct 30 01:17:42.016543 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:42.016504 2573 generic.go:356] "Generic (PLEG): container finished" podID="d205185b-fc45-4a76-9719-20b00107abf1" containerID="fa086018fcdc7e625873376ddc1edd48c8c59518991ac6ffee80dc216b9ef4af" exitCode=0 Oct 30 01:17:42.016677 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:42.016574 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d205185b-fc45-4a76-9719-20b00107abf1","Type":"ContainerDied","Data":"fa086018fcdc7e625873376ddc1edd48c8c59518991ac6ffee80dc216b9ef4af"} Oct 30 01:17:42.018557 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:42.018315 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b6fbb6f6f-9z4ws" event={"ID":"db13cab2-9b76-44b5-8577-9f8f0d3f292e","Type":"ContainerStarted","Data":"2240a9d5917c8ae0ac143c8af0a3020dba0bc4a18ace2943ce2ebba50d98a22f"} Oct 30 01:17:42.018557 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:42.018356 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b6fbb6f6f-9z4ws" event={"ID":"db13cab2-9b76-44b5-8577-9f8f0d3f292e","Type":"ContainerStarted","Data":"223f329baedfd1629bc7e494bc60ac6c4b654bcbc118acaff1ad96b96e5fc52a"} Oct 30 01:17:42.040085 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:42.040052 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/380432b5-86bc-4dd3-a846-d7875d621a36-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-944876b99-29n9f\" (UID: \"380432b5-86bc-4dd3-a846-d7875d621a36\") " pod="openshift-monitoring/metrics-server-944876b99-29n9f" Oct 30 01:17:42.040229 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:42.040203 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/380432b5-86bc-4dd3-a846-d7875d621a36-audit-log\") pod \"metrics-server-944876b99-29n9f\" (UID: \"380432b5-86bc-4dd3-a846-d7875d621a36\") " pod="openshift-monitoring/metrics-server-944876b99-29n9f" Oct 30 01:17:42.040434 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:42.040392 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/380432b5-86bc-4dd3-a846-d7875d621a36-metrics-server-audit-profiles\") pod \"metrics-server-944876b99-29n9f\" (UID: \"380432b5-86bc-4dd3-a846-d7875d621a36\") " pod="openshift-monitoring/metrics-server-944876b99-29n9f" Oct 30 01:17:42.040749 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:42.040728 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/380432b5-86bc-4dd3-a846-d7875d621a36-secret-metrics-server-client-certs\") pod \"metrics-server-944876b99-29n9f\" (UID: \"380432b5-86bc-4dd3-a846-d7875d621a36\") " pod="openshift-monitoring/metrics-server-944876b99-29n9f" Oct 30 01:17:42.040946 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:42.040822 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/380432b5-86bc-4dd3-a846-d7875d621a36-secret-metrics-server-tls\") pod \"metrics-server-944876b99-29n9f\" (UID: \"380432b5-86bc-4dd3-a846-d7875d621a36\") " pod="openshift-monitoring/metrics-server-944876b99-29n9f" Oct 30 01:17:42.041055 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:42.040950 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjqdw\" (UniqueName: \"kubernetes.io/projected/380432b5-86bc-4dd3-a846-d7875d621a36-kube-api-access-wjqdw\") pod \"metrics-server-944876b99-29n9f\" (UID: \"380432b5-86bc-4dd3-a846-d7875d621a36\") " pod="openshift-monitoring/metrics-server-944876b99-29n9f" Oct 30 01:17:42.041172 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:42.041140 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/380432b5-86bc-4dd3-a846-d7875d621a36-client-ca-bundle\") pod \"metrics-server-944876b99-29n9f\" (UID: \"380432b5-86bc-4dd3-a846-d7875d621a36\") " pod="openshift-monitoring/metrics-server-944876b99-29n9f" Oct 30 01:17:42.114622 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:42.114573 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7b6fbb6f6f-9z4ws" podStartSLOduration=1.114558437 podStartE2EDuration="1.114558437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:42.114060864 +0000 UTC m=+57.010038847" watchObservedRunningTime="2026-10-30 01:17:42.114558437 +0000 UTC m=+57.010536420" Oct 30 01:17:42.141545 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:42.141511 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/380432b5-86bc-4dd3-a846-d7875d621a36-audit-log\") pod \"metrics-server-944876b99-29n9f\" (UID: \"380432b5-86bc-4dd3-a846-d7875d621a36\") " pod="openshift-monitoring/metrics-server-944876b99-29n9f" Oct 30 01:17:42.141692 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:42.141586 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/380432b5-86bc-4dd3-a846-d7875d621a36-metrics-server-audit-profiles\") pod \"metrics-server-944876b99-29n9f\" (UID: \"380432b5-86bc-4dd3-a846-d7875d621a36\") " pod="openshift-monitoring/metrics-server-944876b99-29n9f" Oct 30 01:17:42.141692 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:42.141624 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/380432b5-86bc-4dd3-a846-d7875d621a36-secret-metrics-server-client-certs\") pod \"metrics-server-944876b99-29n9f\" (UID: \"380432b5-86bc-4dd3-a846-d7875d621a36\") " pod="openshift-monitoring/metrics-server-944876b99-29n9f" Oct 30 01:17:42.141692 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:42.141646 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/380432b5-86bc-4dd3-a846-d7875d621a36-secret-metrics-server-tls\") pod \"metrics-server-944876b99-29n9f\" (UID: \"380432b5-86bc-4dd3-a846-d7875d621a36\") " pod="openshift-monitoring/metrics-server-944876b99-29n9f" Oct 30 01:17:42.141692 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:42.141676 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjqdw\" (UniqueName: \"kubernetes.io/projected/380432b5-86bc-4dd3-a846-d7875d621a36-kube-api-access-wjqdw\") pod \"metrics-server-944876b99-29n9f\" (UID: \"380432b5-86bc-4dd3-a846-d7875d621a36\") " pod="openshift-monitoring/metrics-server-944876b99-29n9f" Oct 30 01:17:42.141884 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:42.141720 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/380432b5-86bc-4dd3-a846-d7875d621a36-client-ca-bundle\") pod \"metrics-server-944876b99-29n9f\" (UID: \"380432b5-86bc-4dd3-a846-d7875d621a36\") " pod="openshift-monitoring/metrics-server-944876b99-29n9f" Oct 30 01:17:42.141884 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:42.141760 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/380432b5-86bc-4dd3-a846-d7875d621a36-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-944876b99-29n9f\" (UID: \"380432b5-86bc-4dd3-a846-d7875d621a36\") " pod="openshift-monitoring/metrics-server-944876b99-29n9f" Oct 30 01:17:42.142036 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:42.142010 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/380432b5-86bc-4dd3-a846-d7875d621a36-audit-log\") pod \"metrics-server-944876b99-29n9f\" (UID: \"380432b5-86bc-4dd3-a846-d7875d621a36\") " pod="openshift-monitoring/metrics-server-944876b99-29n9f" Oct 30 01:17:42.142451 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:42.142423 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/380432b5-86bc-4dd3-a846-d7875d621a36-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-944876b99-29n9f\" (UID: \"380432b5-86bc-4dd3-a846-d7875d621a36\") " pod="openshift-monitoring/metrics-server-944876b99-29n9f" Oct 30 01:17:42.142667 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:42.142650 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/380432b5-86bc-4dd3-a846-d7875d621a36-metrics-server-audit-profiles\") pod \"metrics-server-944876b99-29n9f\" (UID: \"380432b5-86bc-4dd3-a846-d7875d621a36\") " pod="openshift-monitoring/metrics-server-944876b99-29n9f" Oct 30 01:17:42.144025 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:42.143997 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/380432b5-86bc-4dd3-a846-d7875d621a36-client-ca-bundle\") pod \"metrics-server-944876b99-29n9f\" (UID: \"380432b5-86bc-4dd3-a846-d7875d621a36\") " pod="openshift-monitoring/metrics-server-944876b99-29n9f" Oct 30 01:17:42.144100 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:42.144036 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/380432b5-86bc-4dd3-a846-d7875d621a36-secret-metrics-server-client-certs\") pod \"metrics-server-944876b99-29n9f\" (UID: \"380432b5-86bc-4dd3-a846-d7875d621a36\") " pod="openshift-monitoring/metrics-server-944876b99-29n9f" Oct 30 01:17:42.144100 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:42.144084 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/380432b5-86bc-4dd3-a846-d7875d621a36-secret-metrics-server-tls\") pod \"metrics-server-944876b99-29n9f\" (UID: \"380432b5-86bc-4dd3-a846-d7875d621a36\") " pod="openshift-monitoring/metrics-server-944876b99-29n9f" Oct 30 01:17:42.161470 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:42.161448 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjqdw\" (UniqueName: \"kubernetes.io/projected/380432b5-86bc-4dd3-a846-d7875d621a36-kube-api-access-wjqdw\") pod \"metrics-server-944876b99-29n9f\" (UID: \"380432b5-86bc-4dd3-a846-d7875d621a36\") " pod="openshift-monitoring/metrics-server-944876b99-29n9f" Oct 30 01:17:42.295849 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:42.295817 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-944876b99-29n9f" Oct 30 01:17:42.604729 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:42.604652 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-944876b99-29n9f"] Oct 30 01:17:42.608399 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:17:42.608375 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod380432b5_86bc_4dd3_a846_d7875d621a36.slice/crio-b92a8a76a7ebb37f3fe475c4ebcaa5d33d0f373c92559c7428367ddf74abe492 WatchSource:0}: Error finding container b92a8a76a7ebb37f3fe475c4ebcaa5d33d0f373c92559c7428367ddf74abe492: Status 404 returned error can't find the container with id b92a8a76a7ebb37f3fe475c4ebcaa5d33d0f373c92559c7428367ddf74abe492 Oct 30 01:17:42.978234 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:42.978155 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-cwzvk" Oct 30 01:17:43.022859 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:43.022828 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-944876b99-29n9f" event={"ID":"380432b5-86bc-4dd3-a846-d7875d621a36","Type":"ContainerStarted","Data":"b92a8a76a7ebb37f3fe475c4ebcaa5d33d0f373c92559c7428367ddf74abe492"} Oct 30 01:17:44.032208 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:44.032171 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d205185b-fc45-4a76-9719-20b00107abf1","Type":"ContainerStarted","Data":"faecbc4dc56aabdfd4bddaf0013c0303a9c2ca18ac5b800ae1bdcff268fa6d94"} Oct 30 01:17:44.032676 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:44.032284 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d205185b-fc45-4a76-9719-20b00107abf1","Type":"ContainerStarted","Data":"2f312c75cad0960abe3ae70749c0cbae1dc1a744e61c9619efc01a84f6cca31b"} Oct 30 01:17:44.032676 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:44.032330 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d205185b-fc45-4a76-9719-20b00107abf1","Type":"ContainerStarted","Data":"5e56bb5fc49748e897907a5d67bde94bb556b6c08dcd80e36eb70b7bcb98ee74"} Oct 30 01:17:44.032676 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:44.032377 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d205185b-fc45-4a76-9719-20b00107abf1","Type":"ContainerStarted","Data":"087568965eb285b660987b3f2418bff23f91bd55c25f5c9b6abc10903538f624"} Oct 30 01:17:44.032676 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:44.032416 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d205185b-fc45-4a76-9719-20b00107abf1","Type":"ContainerStarted","Data":"7c4f99037568d2ad4c8bac57b44357ea2bef68911bc930f4dd8ff2301cb02b04"} Oct 30 01:17:45.040551 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:45.040501 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d205185b-fc45-4a76-9719-20b00107abf1","Type":"ContainerStarted","Data":"b7e6935898f0ee20cc221ace812df50f79bc1400474593d50e2ce45b44280d47"} Oct 30 01:17:45.041978 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:45.041953 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-944876b99-29n9f" event={"ID":"380432b5-86bc-4dd3-a846-d7875d621a36","Type":"ContainerStarted","Data":"41a1a1aae3fab931bb4badbb99e0ada5aeb746068a37bc42e55b78a4e273569f"} Oct 30 01:17:45.137138 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:45.137086 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.833980762 podStartE2EDuration="7.137069829s" totalImagesPullingTime="3.291037636s" totalInitContainerRuntime="12.051431ms" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:45.123183545 +0000 UTC m=+60.019161528" watchObservedRunningTime="2026-10-30 01:17:45.137069829 +0000 UTC m=+60.033047828" Oct 30 01:17:45.183303 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:45.183250 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-944876b99-29n9f" podStartSLOduration=2.6855068380000002 podStartE2EDuration="4.183235695s" totalImagesPullingTime="1.497728857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:45.182283206 +0000 UTC m=+60.078261189" watchObservedRunningTime="2026-10-30 01:17:45.183235695 +0000 UTC m=+60.079213741" Oct 30 01:17:50.123216 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:50.123181 2573 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b6fbb6f6f-9z4ws"] Oct 30 01:17:50.155465 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:50.155433 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-9477b4757-pfqgz"] Oct 30 01:17:50.159803 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:50.159785 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9477b4757-pfqgz" Oct 30 01:17:50.179402 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:50.179375 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9477b4757-pfqgz"] Oct 30 01:17:50.214243 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:50.214210 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d834c2a-f008-492e-9aac-362fe7a73672-console-oauth-config\") pod \"console-9477b4757-pfqgz\" (UID: \"7d834c2a-f008-492e-9aac-362fe7a73672\") " pod="openshift-console/console-9477b4757-pfqgz" Oct 30 01:17:50.214243 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:50.214244 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7d834c2a-f008-492e-9aac-362fe7a73672-console-config\") pod \"console-9477b4757-pfqgz\" (UID: \"7d834c2a-f008-492e-9aac-362fe7a73672\") " pod="openshift-console/console-9477b4757-pfqgz" Oct 30 01:17:50.214436 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:50.214341 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d834c2a-f008-492e-9aac-362fe7a73672-oauth-serving-cert\") pod \"console-9477b4757-pfqgz\" (UID: \"7d834c2a-f008-492e-9aac-362fe7a73672\") " pod="openshift-console/console-9477b4757-pfqgz" Oct 30 01:17:50.214436 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:50.214415 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d834c2a-f008-492e-9aac-362fe7a73672-service-ca\") pod \"console-9477b4757-pfqgz\" (UID: \"7d834c2a-f008-492e-9aac-362fe7a73672\") " pod="openshift-console/console-9477b4757-pfqgz" Oct 30 01:17:50.214551 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:50.214443 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d834c2a-f008-492e-9aac-362fe7a73672-console-serving-cert\") pod \"console-9477b4757-pfqgz\" (UID: \"7d834c2a-f008-492e-9aac-362fe7a73672\") " pod="openshift-console/console-9477b4757-pfqgz" Oct 30 01:17:50.214551 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:50.214478 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d834c2a-f008-492e-9aac-362fe7a73672-tmp\") pod \"console-9477b4757-pfqgz\" (UID: \"7d834c2a-f008-492e-9aac-362fe7a73672\") " pod="openshift-console/console-9477b4757-pfqgz" Oct 30 01:17:50.214551 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:50.214494 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d834c2a-f008-492e-9aac-362fe7a73672-trusted-ca-bundle\") pod \"console-9477b4757-pfqgz\" (UID: \"7d834c2a-f008-492e-9aac-362fe7a73672\") " pod="openshift-console/console-9477b4757-pfqgz" Oct 30 01:17:50.214689 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:50.214568 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd9jh\" (UniqueName: \"kubernetes.io/projected/7d834c2a-f008-492e-9aac-362fe7a73672-kube-api-access-sd9jh\") pod \"console-9477b4757-pfqgz\" (UID: \"7d834c2a-f008-492e-9aac-362fe7a73672\") " pod="openshift-console/console-9477b4757-pfqgz" Oct 30 01:17:50.315266 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:50.315227 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d834c2a-f008-492e-9aac-362fe7a73672-console-oauth-config\") pod \"console-9477b4757-pfqgz\" (UID: \"7d834c2a-f008-492e-9aac-362fe7a73672\") " pod="openshift-console/console-9477b4757-pfqgz" Oct 30 01:17:50.315266 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:50.315261 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7d834c2a-f008-492e-9aac-362fe7a73672-console-config\") pod \"console-9477b4757-pfqgz\" (UID: \"7d834c2a-f008-492e-9aac-362fe7a73672\") " pod="openshift-console/console-9477b4757-pfqgz" Oct 30 01:17:50.315474 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:50.315294 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d834c2a-f008-492e-9aac-362fe7a73672-oauth-serving-cert\") pod \"console-9477b4757-pfqgz\" (UID: \"7d834c2a-f008-492e-9aac-362fe7a73672\") " pod="openshift-console/console-9477b4757-pfqgz" Oct 30 01:17:50.315474 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:50.315326 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d834c2a-f008-492e-9aac-362fe7a73672-service-ca\") pod \"console-9477b4757-pfqgz\" (UID: \"7d834c2a-f008-492e-9aac-362fe7a73672\") " pod="openshift-console/console-9477b4757-pfqgz" Oct 30 01:17:50.315474 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:50.315343 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d834c2a-f008-492e-9aac-362fe7a73672-console-serving-cert\") pod \"console-9477b4757-pfqgz\" (UID: \"7d834c2a-f008-492e-9aac-362fe7a73672\") " pod="openshift-console/console-9477b4757-pfqgz" Oct 30 01:17:50.315474 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:50.315357 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d834c2a-f008-492e-9aac-362fe7a73672-tmp\") pod \"console-9477b4757-pfqgz\" (UID: \"7d834c2a-f008-492e-9aac-362fe7a73672\") " pod="openshift-console/console-9477b4757-pfqgz" Oct 30 01:17:50.315474 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:50.315375 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d834c2a-f008-492e-9aac-362fe7a73672-trusted-ca-bundle\") pod \"console-9477b4757-pfqgz\" (UID: \"7d834c2a-f008-492e-9aac-362fe7a73672\") " pod="openshift-console/console-9477b4757-pfqgz" Oct 30 01:17:50.315474 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:50.315404 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd9jh\" (UniqueName: \"kubernetes.io/projected/7d834c2a-f008-492e-9aac-362fe7a73672-kube-api-access-sd9jh\") pod \"console-9477b4757-pfqgz\" (UID: \"7d834c2a-f008-492e-9aac-362fe7a73672\") " pod="openshift-console/console-9477b4757-pfqgz" Oct 30 01:17:50.315817 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:50.315786 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d834c2a-f008-492e-9aac-362fe7a73672-tmp\") pod \"console-9477b4757-pfqgz\" (UID: \"7d834c2a-f008-492e-9aac-362fe7a73672\") " pod="openshift-console/console-9477b4757-pfqgz" Oct 30 01:17:50.316108 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:50.316082 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7d834c2a-f008-492e-9aac-362fe7a73672-console-config\") pod \"console-9477b4757-pfqgz\" (UID: \"7d834c2a-f008-492e-9aac-362fe7a73672\") " pod="openshift-console/console-9477b4757-pfqgz" Oct 30 01:17:50.316317 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:50.316294 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d834c2a-f008-492e-9aac-362fe7a73672-service-ca\") pod \"console-9477b4757-pfqgz\" (UID: \"7d834c2a-f008-492e-9aac-362fe7a73672\") " pod="openshift-console/console-9477b4757-pfqgz" Oct 30 01:17:50.316361 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:50.316332 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d834c2a-f008-492e-9aac-362fe7a73672-oauth-serving-cert\") pod \"console-9477b4757-pfqgz\" (UID: \"7d834c2a-f008-492e-9aac-362fe7a73672\") " pod="openshift-console/console-9477b4757-pfqgz" Oct 30 01:17:50.316770 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:50.316737 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d834c2a-f008-492e-9aac-362fe7a73672-trusted-ca-bundle\") pod \"console-9477b4757-pfqgz\" (UID: \"7d834c2a-f008-492e-9aac-362fe7a73672\") " pod="openshift-console/console-9477b4757-pfqgz" Oct 30 01:17:50.318062 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:50.318038 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d834c2a-f008-492e-9aac-362fe7a73672-console-oauth-config\") pod \"console-9477b4757-pfqgz\" (UID: \"7d834c2a-f008-492e-9aac-362fe7a73672\") " pod="openshift-console/console-9477b4757-pfqgz" Oct 30 01:17:50.318153 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:50.318090 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d834c2a-f008-492e-9aac-362fe7a73672-console-serving-cert\") pod \"console-9477b4757-pfqgz\" (UID: \"7d834c2a-f008-492e-9aac-362fe7a73672\") " pod="openshift-console/console-9477b4757-pfqgz" Oct 30 01:17:50.337834 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:50.337806 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd9jh\" (UniqueName: \"kubernetes.io/projected/7d834c2a-f008-492e-9aac-362fe7a73672-kube-api-access-sd9jh\") pod \"console-9477b4757-pfqgz\" (UID: \"7d834c2a-f008-492e-9aac-362fe7a73672\") " pod="openshift-console/console-9477b4757-pfqgz" Oct 30 01:17:50.416245 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:50.416169 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94864d90-1dd8-472e-a414-29c6b6a4f8f6-metrics-certs\") pod \"network-metrics-daemon-xj8mh\" (UID: \"94864d90-1dd8-472e-a414-29c6b6a4f8f6\") " pod="openshift-multus/network-metrics-daemon-xj8mh" Oct 30 01:17:50.416380 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:50.416339 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f10aab9c-d4e6-45b5-a703-146114a31329-original-pull-secret\") pod \"global-pull-secret-syncer-dxz45\" (UID: \"f10aab9c-d4e6-45b5-a703-146114a31329\") " pod="kube-system/global-pull-secret-syncer-dxz45" Oct 30 01:17:50.431075 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:50.431049 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 01:17:50.431203 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:50.431085 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 01:17:50.438488 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:50.438465 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94864d90-1dd8-472e-a414-29c6b6a4f8f6-metrics-certs\") pod \"network-metrics-daemon-xj8mh\" (UID: \"94864d90-1dd8-472e-a414-29c6b6a4f8f6\") " pod="openshift-multus/network-metrics-daemon-xj8mh" Oct 30 01:17:50.438628 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:50.438562 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f10aab9c-d4e6-45b5-a703-146114a31329-original-pull-secret\") pod \"global-pull-secret-syncer-dxz45\" (UID: \"f10aab9c-d4e6-45b5-a703-146114a31329\") " pod="kube-system/global-pull-secret-syncer-dxz45" Oct 30 01:17:50.470297 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:50.470267 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9477b4757-pfqgz" Oct 30 01:17:50.517085 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:50.517052 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89k2s\" (UniqueName: \"kubernetes.io/projected/f4602ac2-652b-4026-9a0d-d72162bb3258-kube-api-access-89k2s\") pod \"network-check-target-bbhx5\" (UID: \"f4602ac2-652b-4026-9a0d-d72162bb3258\") " pod="openshift-network-diagnostics/network-check-target-bbhx5" Oct 30 01:17:50.532068 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:50.532040 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 01:17:50.552697 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:50.552676 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 01:17:50.560841 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:50.560819 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89k2s\" (UniqueName: \"kubernetes.io/projected/f4602ac2-652b-4026-9a0d-d72162bb3258-kube-api-access-89k2s\") pod \"network-check-target-bbhx5\" (UID: \"f4602ac2-652b-4026-9a0d-d72162bb3258\") " pod="openshift-network-diagnostics/network-check-target-bbhx5" Oct 30 01:17:50.663300 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:50.663268 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xp5gx\"" Oct 30 01:17:50.669418 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:50.669341 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-xj8mh" Oct 30 01:17:50.751110 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:50.750510 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-n79rv\"" Oct 30 01:17:50.756584 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:50.756554 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dxz45" Oct 30 01:17:50.760701 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:50.760683 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-wjscp\"" Oct 30 01:17:50.766795 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:50.766759 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bbhx5" Oct 30 01:17:50.772901 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:50.772835 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9477b4757-pfqgz"] Oct 30 01:17:50.777917 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:17:50.777891 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d834c2a_f008_492e_9aac_362fe7a73672.slice/crio-a095b5dc32b2cf300fed1b5e01fe9e5bb6ede4318878b8166dfe02367469e34c WatchSource:0}: Error finding container a095b5dc32b2cf300fed1b5e01fe9e5bb6ede4318878b8166dfe02367469e34c: Status 404 returned error can't find the container with id a095b5dc32b2cf300fed1b5e01fe9e5bb6ede4318878b8166dfe02367469e34c Oct 30 01:17:50.972015 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:50.968313 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-xj8mh"] Oct 30 01:17:50.972015 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:17:50.971744 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94864d90_1dd8_472e_a414_29c6b6a4f8f6.slice/crio-3a6ccb5dd4cd9de9186c36c98eca4a4c4604c4435b29e1e08b52832bff369331 WatchSource:0}: Error finding container 3a6ccb5dd4cd9de9186c36c98eca4a4c4604c4435b29e1e08b52832bff369331: Status 404 returned error can't find the container with id 3a6ccb5dd4cd9de9186c36c98eca4a4c4604c4435b29e1e08b52832bff369331 Oct 30 01:17:51.064596 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:51.064561 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dxz45"] Oct 30 01:17:51.067905 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:51.067873 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xj8mh" event={"ID":"94864d90-1dd8-472e-a414-29c6b6a4f8f6","Type":"ContainerStarted","Data":"3a6ccb5dd4cd9de9186c36c98eca4a4c4604c4435b29e1e08b52832bff369331"} Oct 30 01:17:51.069646 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:17:51.069619 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf10aab9c_d4e6_45b5_a703_146114a31329.slice/crio-e44db52d977777a8d97795e0523bae26cc5c1d0767befa45718e4c5062f195b1 WatchSource:0}: Error finding container e44db52d977777a8d97795e0523bae26cc5c1d0767befa45718e4c5062f195b1: Status 404 returned error can't find the container with id e44db52d977777a8d97795e0523bae26cc5c1d0767befa45718e4c5062f195b1 Oct 30 01:17:51.070466 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:51.070374 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9477b4757-pfqgz" event={"ID":"7d834c2a-f008-492e-9aac-362fe7a73672","Type":"ContainerStarted","Data":"81eb36f8d4fbb810a26e5d10f9f78920fed632af86ea4dac4f5eebbc9fa52ee4"} Oct 30 01:17:51.070466 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:51.070417 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9477b4757-pfqgz" event={"ID":"7d834c2a-f008-492e-9aac-362fe7a73672","Type":"ContainerStarted","Data":"a095b5dc32b2cf300fed1b5e01fe9e5bb6ede4318878b8166dfe02367469e34c"} Oct 30 01:17:51.093791 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:51.093763 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bbhx5"] Oct 30 01:17:51.097235 ip-10-0-135-65 kubenswrapper[2573]: W1030 01:17:51.097211 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4602ac2_652b_4026_9a0d_d72162bb3258.slice/crio-3458374121b40d6bc0a516989f33c9354e56a50e5c95d56a86ca6dc6b5d22ff6 WatchSource:0}: Error finding container 3458374121b40d6bc0a516989f33c9354e56a50e5c95d56a86ca6dc6b5d22ff6: Status 404 returned error can't find the container with id 3458374121b40d6bc0a516989f33c9354e56a50e5c95d56a86ca6dc6b5d22ff6 Oct 30 01:17:51.129501 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:51.129453 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-9477b4757-pfqgz" podStartSLOduration=1.129425289 podStartE2EDuration="1.129425289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:17:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:51.129191701 +0000 UTC m=+66.025169685" watchObservedRunningTime="2026-10-30 01:17:51.129425289 +0000 UTC m=+66.025403272" Oct 30 01:17:51.414583 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:51.414340 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b6fbb6f6f-9z4ws" Oct 30 01:17:52.085606 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:52.085494 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dxz45" event={"ID":"f10aab9c-d4e6-45b5-a703-146114a31329","Type":"ContainerStarted","Data":"e44db52d977777a8d97795e0523bae26cc5c1d0767befa45718e4c5062f195b1"} Oct 30 01:17:52.088615 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:52.088565 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bbhx5" event={"ID":"f4602ac2-652b-4026-9a0d-d72162bb3258","Type":"ContainerStarted","Data":"3458374121b40d6bc0a516989f33c9354e56a50e5c95d56a86ca6dc6b5d22ff6"} Oct 30 01:17:53.095497 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:53.095453 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xj8mh" event={"ID":"94864d90-1dd8-472e-a414-29c6b6a4f8f6","Type":"ContainerStarted","Data":"c956f7e3d11c79fbfbf1bfd20f2a1c8dfa0a551941e6ae7fe88fe6ae70d80691"} Oct 30 01:17:53.095925 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:53.095517 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-xj8mh" event={"ID":"94864d90-1dd8-472e-a414-29c6b6a4f8f6","Type":"ContainerStarted","Data":"c5d0218546125ca460a683514fb27a1b34039d80c5ceb61f2f106f6e2afc9b85"} Oct 30 01:17:53.136918 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:53.136852 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-xj8mh" podStartSLOduration=66.655671347 podStartE2EDuration="1m8.136833964s" totalImagesPullingTime="1.481162617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:53.135007238 +0000 UTC m=+68.030985242" watchObservedRunningTime="2026-10-30 01:17:53.136833964 +0000 UTC m=+68.032811950" Oct 30 01:17:55.106685 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:55.106645 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bbhx5" event={"ID":"f4602ac2-652b-4026-9a0d-d72162bb3258","Type":"ContainerStarted","Data":"f2b75de2838a216cd3d6307c7a9f94ace1bb06a8e9bfbd48e6bb118b535e136f"} Oct 30 01:17:55.107159 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:55.106906 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bbhx5" Oct 30 01:17:55.148023 ip-10-0-135-65 kubenswrapper[2573]: I1030 01:17:55.147975 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bbhx5" podStartSLOduration=66.540232377 podStartE2EDuration="1m10.147960539s" totalImagesPullingTime="3.607728162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:16:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:17:55.146890276 +0000 UTC m=+70.042868261" watchObservedRunningTime="2026-10-30 01:17:55.147960539 +0000 UTC m=+70.043938522" Oct 30 01:17:56.369495 ip-10-0-135-65 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 01:17:56.380913 ip-10-0-135-65 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 01:17:56.381075 ip-10-0-135-65 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 01:17:56.381126 ip-10-0-135-65 systemd[1]: kubelet.service: Consumed 3.934s CPU time. Oct 30 01:17:56.391214 ip-10-0-135-65 systemd[1]: Starting Kubernetes Kubelet... Oct 30 01:17:56.452203 ip-10-0-135-65 kubenswrapper[6221]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 01:17:56.452203 ip-10-0-135-65 kubenswrapper[6221]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 01:17:56.452203 ip-10-0-135-65 kubenswrapper[6221]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 01:17:56.452203 ip-10-0-135-65 kubenswrapper[6221]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 01:17:56.454547 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454444 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:17:56.454547 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454548 6221 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:17:56.454547 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454555 6221 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:17:56.454776 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454559 6221 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:17:56.454776 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454564 6221 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:17:56.454776 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454568 6221 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:17:56.454776 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454572 6221 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:17:56.454776 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454576 6221 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:17:56.454776 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454580 6221 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:17:56.454776 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454584 6221 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:17:56.454776 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454588 6221 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:17:56.454776 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454593 6221 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:17:56.454776 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454597 6221 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:17:56.454776 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454601 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:17:56.454776 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454605 6221 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:17:56.454776 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454610 6221 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:17:56.454776 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454613 6221 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:17:56.454776 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454618 6221 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:17:56.454776 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454622 6221 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:17:56.454776 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454626 6221 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:17:56.454776 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454630 6221 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:17:56.454776 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454634 6221 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:17:56.455636 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454638 6221 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:17:56.455636 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454642 6221 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:17:56.455636 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454646 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:17:56.455636 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454650 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:17:56.455636 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454654 6221 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:17:56.455636 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454668 6221 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:17:56.455636 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454673 6221 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:17:56.455636 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454678 6221 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:17:56.455636 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454683 6221 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:17:56.455636 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454687 6221 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:17:56.455636 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454691 6221 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:17:56.455636 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454696 6221 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:17:56.455636 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454700 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:17:56.455636 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454705 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:17:56.455636 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454709 6221 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:17:56.455636 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454713 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:17:56.455636 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454718 6221 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:17:56.455636 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454722 6221 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:17:56.455636 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454727 6221 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:17:56.455636 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454731 6221 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:17:56.456511 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454735 6221 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:17:56.456511 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454739 6221 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:17:56.456511 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454744 6221 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:17:56.456511 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454749 6221 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:17:56.456511 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454753 6221 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:17:56.456511 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454758 6221 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:17:56.456511 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454762 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:17:56.456511 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454767 6221 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:17:56.456511 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454771 6221 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:17:56.456511 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454775 6221 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:17:56.456511 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454779 6221 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:17:56.456511 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454784 6221 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:17:56.456511 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454789 6221 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:17:56.456511 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454793 6221 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:17:56.456511 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454797 6221 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:17:56.456511 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454802 6221 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:17:56.456511 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454806 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:17:56.456511 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454811 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:17:56.456511 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454820 6221 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:17:56.457357 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454828 6221 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:17:56.457357 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454833 6221 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:17:56.457357 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454837 6221 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:17:56.457357 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454842 6221 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:17:56.457357 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454846 6221 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:17:56.457357 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454851 6221 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:17:56.457357 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454857 6221 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:17:56.457357 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454863 6221 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:17:56.457357 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454869 6221 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:17:56.457357 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454874 6221 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:17:56.457357 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454878 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:17:56.457357 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454883 6221 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:17:56.457357 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454887 6221 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:17:56.457357 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454892 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:17:56.457357 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454896 6221 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:17:56.457357 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454900 6221 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:17:56.457357 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454905 6221 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:17:56.457357 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454910 6221 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:17:56.457357 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454914 6221 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:17:56.458292 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454919 6221 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:17:56.458292 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454923 6221 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:17:56.458292 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454927 6221 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:17:56.458292 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454931 6221 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:17:56.458292 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454936 6221 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:17:56.458292 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454947 6221 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:17:56.458292 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454952 6221 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:17:56.458292 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454956 6221 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:17:56.458292 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454961 6221 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:17:56.458292 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454965 6221 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:17:56.458292 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454969 6221 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:17:56.458292 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454974 6221 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:17:56.458292 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454979 6221 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:17:56.458292 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454989 6221 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:17:56.458292 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454994 6221 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:17:56.458292 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.454999 6221 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:17:56.458292 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.455004 6221 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:17:56.458292 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.455008 6221 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:17:56.458292 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.455014 6221 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:17:56.458292 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.455018 6221 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:17:56.459218 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.455023 6221 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:17:56.459218 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.455027 6221 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:17:56.459218 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.455031 6221 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:17:56.459218 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.455036 6221 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:17:56.459218 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.455040 6221 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:17:56.459218 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.455045 6221 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:17:56.459218 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456283 6221 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:17:56.459218 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456294 6221 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:17:56.459218 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456300 6221 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:17:56.459218 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456305 6221 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:17:56.459218 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456310 6221 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:17:56.459218 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456314 6221 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:17:56.459218 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456319 6221 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:17:56.459218 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456324 6221 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:17:56.459218 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456328 6221 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:17:56.459218 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456333 6221 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:17:56.459218 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456337 6221 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:17:56.459218 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456341 6221 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:17:56.459218 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456346 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:17:56.460086 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456351 6221 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:17:56.460086 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456355 6221 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:17:56.460086 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456360 6221 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:17:56.460086 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456364 6221 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:17:56.460086 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456368 6221 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:17:56.460086 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456374 6221 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:17:56.460086 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456378 6221 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:17:56.460086 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456383 6221 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:17:56.460086 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456388 6221 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:17:56.460086 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456392 6221 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:17:56.460086 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456397 6221 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:17:56.460086 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456401 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:17:56.460086 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456406 6221 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:17:56.460086 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456410 6221 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:17:56.460086 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456414 6221 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:17:56.460086 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456419 6221 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:17:56.460086 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456423 6221 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:17:56.460086 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456427 6221 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:17:56.460086 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456431 6221 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:17:56.460086 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456436 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:17:56.460964 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456440 6221 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:17:56.460964 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456445 6221 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:17:56.460964 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456450 6221 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:17:56.460964 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456454 6221 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:17:56.460964 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456459 6221 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:17:56.460964 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456463 6221 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:17:56.460964 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456468 6221 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:17:56.460964 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456472 6221 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:17:56.460964 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456476 6221 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:17:56.460964 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456481 6221 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:17:56.460964 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456485 6221 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:17:56.460964 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456490 6221 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:17:56.460964 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456494 6221 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:17:56.460964 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456498 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:17:56.460964 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456502 6221 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:17:56.460964 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456506 6221 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:17:56.460964 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456510 6221 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:17:56.460964 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456515 6221 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:17:56.460964 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456519 6221 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:17:56.460964 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456523 6221 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:17:56.461623 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456545 6221 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:17:56.461623 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456550 6221 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:17:56.461623 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456554 6221 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:17:56.461623 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456559 6221 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:17:56.461623 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456563 6221 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:17:56.461623 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456568 6221 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:17:56.461623 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456573 6221 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:17:56.461623 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456577 6221 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:17:56.461623 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456581 6221 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:17:56.461623 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456586 6221 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:17:56.461623 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456590 6221 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:17:56.461623 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456595 6221 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:17:56.461623 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456599 6221 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:17:56.461623 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456604 6221 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:17:56.461623 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456608 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:17:56.461623 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456615 6221 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:17:56.461623 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456623 6221 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:17:56.461623 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456628 6221 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:17:56.461623 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456634 6221 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:17:56.462088 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456639 6221 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:17:56.462088 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456644 6221 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:17:56.462088 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456648 6221 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:17:56.462088 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456652 6221 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:17:56.462088 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456657 6221 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:17:56.462088 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456664 6221 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:17:56.462088 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456669 6221 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:17:56.462088 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456675 6221 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:17:56.462088 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456681 6221 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:17:56.462088 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456685 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:17:56.462088 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456690 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:17:56.462088 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456694 6221 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:17:56.462088 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456699 6221 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:17:56.462088 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456703 6221 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:17:56.462088 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456709 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:17:56.462088 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456713 6221 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:17:56.462088 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456717 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:17:56.462088 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456722 6221 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:17:56.462088 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456726 6221 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:17:56.462571 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456730 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:17:56.462571 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456735 6221 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:17:56.462571 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456740 6221 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:17:56.462571 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456744 6221 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:17:56.462571 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456749 6221 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:17:56.462571 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456753 6221 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:17:56.462571 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456758 6221 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:17:56.462571 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456762 6221 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:17:56.462571 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456766 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:17:56.462571 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456771 6221 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:17:56.462571 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456777 6221 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:17:56.462571 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456781 6221 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:17:56.462571 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456785 6221 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:17:56.462571 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456790 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:17:56.462571 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.456794 6221 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:17:56.462571 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457671 6221 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:17:56.462571 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457681 6221 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:17:56.462571 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457686 6221 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:17:56.462571 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457691 6221 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:17:56.462571 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457696 6221 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:17:56.463113 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457701 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:17:56.463113 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457706 6221 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:17:56.463113 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457710 6221 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:17:56.463113 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457714 6221 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:17:56.463113 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457718 6221 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:17:56.463113 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457723 6221 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:17:56.463113 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457727 6221 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:17:56.463113 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457740 6221 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:17:56.463113 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457746 6221 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:17:56.463113 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457750 6221 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:17:56.463113 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457755 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:17:56.463113 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457759 6221 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:17:56.463113 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457763 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:17:56.463113 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457768 6221 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:17:56.463113 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457772 6221 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:17:56.463113 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457776 6221 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:17:56.463113 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457781 6221 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:17:56.463113 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457785 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:17:56.463113 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457790 6221 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:17:56.463598 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457809 6221 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:17:56.463598 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457813 6221 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:17:56.463598 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457817 6221 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:17:56.463598 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457822 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:17:56.463598 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457826 6221 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:17:56.463598 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457831 6221 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:17:56.463598 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457835 6221 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:17:56.463598 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457839 6221 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:17:56.463598 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457844 6221 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:17:56.463598 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457848 6221 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:17:56.463598 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457852 6221 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:17:56.463598 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457857 6221 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:17:56.463598 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457861 6221 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:17:56.463598 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457865 6221 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:17:56.463598 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457870 6221 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:17:56.463598 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457874 6221 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:17:56.463598 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457878 6221 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:17:56.463598 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457884 6221 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:17:56.463598 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457889 6221 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:17:56.463598 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457893 6221 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:17:56.464105 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457897 6221 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:17:56.464105 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457909 6221 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:17:56.464105 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457913 6221 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:17:56.464105 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457918 6221 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:17:56.464105 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457923 6221 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:17:56.464105 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457927 6221 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:17:56.464105 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457931 6221 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:17:56.464105 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457935 6221 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:17:56.464105 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457940 6221 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:17:56.464105 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457945 6221 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:17:56.464105 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457949 6221 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:17:56.464105 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457953 6221 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:17:56.464105 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457959 6221 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:17:56.464105 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457964 6221 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:17:56.464105 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457968 6221 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:17:56.464105 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457973 6221 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:17:56.464105 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457977 6221 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:17:56.464105 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457981 6221 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:17:56.464105 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457985 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:17:56.464105 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457990 6221 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:17:56.464632 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457994 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:17:56.464632 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.457999 6221 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:17:56.464632 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.458005 6221 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:17:56.464632 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.458011 6221 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:17:56.464632 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.458017 6221 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:17:56.464632 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.458021 6221 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:17:56.464632 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.458026 6221 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:17:56.464632 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.458031 6221 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:17:56.464632 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.458035 6221 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:17:56.464632 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.458039 6221 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:17:56.464632 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.458044 6221 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:17:56.464632 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.458048 6221 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:17:56.464632 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.458053 6221 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:17:56.464632 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.458057 6221 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:17:56.464632 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.458072 6221 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:17:56.464632 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.458077 6221 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:17:56.464632 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.458082 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:17:56.464632 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.458086 6221 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:17:56.464632 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.458090 6221 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:17:56.465092 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.458094 6221 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:17:56.465092 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.458099 6221 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:17:56.465092 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.458103 6221 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:17:56.465092 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.458108 6221 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:17:56.465092 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.458112 6221 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:17:56.465092 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.458119 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:17:56.465092 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.458123 6221 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:17:56.465092 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.458128 6221 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:17:56.465092 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.458132 6221 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:17:56.465092 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.458136 6221 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:17:56.465092 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.458141 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:17:56.465092 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.458146 6221 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:17:56.465092 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.458150 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:17:56.465092 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.458156 6221 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:17:56.465092 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.458161 6221 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:17:56.465092 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.458165 6221 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:17:56.465092 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.458169 6221 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:17:56.465092 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.458173 6221 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:17:56.465092 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.458178 6221 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:17:56.465821 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.458183 6221 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:17:56.465821 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.458187 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:17:56.465821 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.458191 6221 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:17:56.465821 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.458198 6221 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:17:56.465821 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458327 6221 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 01:17:56.465821 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458338 6221 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 01:17:56.465821 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458351 6221 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 01:17:56.465821 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458358 6221 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 01:17:56.465821 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458365 6221 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 01:17:56.465821 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458387 6221 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 01:17:56.465821 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458395 6221 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 01:17:56.465821 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458401 6221 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 01:17:56.465821 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458407 6221 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 01:17:56.465821 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458412 6221 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 01:17:56.465821 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458418 6221 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 01:17:56.465821 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458423 6221 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 01:17:56.465821 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458428 6221 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 01:17:56.465821 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458433 6221 flags.go:64] FLAG: --cgroup-root="" Oct 30 01:17:56.465821 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458440 6221 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 01:17:56.465821 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458445 6221 flags.go:64] FLAG: --client-ca-file="" Oct 30 01:17:56.465821 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458449 6221 flags.go:64] FLAG: --cloud-provider="external" Oct 30 01:17:56.465821 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458454 6221 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 01:17:56.465821 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458466 6221 flags.go:64] FLAG: --cluster-domain="" Oct 30 01:17:56.466735 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458471 6221 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 01:17:56.466735 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458476 6221 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 01:17:56.466735 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458481 6221 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 01:17:56.466735 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458492 6221 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 01:17:56.466735 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458500 6221 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 01:17:56.466735 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458505 6221 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 01:17:56.466735 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458510 6221 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 01:17:56.466735 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458515 6221 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 01:17:56.466735 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458520 6221 flags.go:64] FLAG: --contention-profiling="false" Oct 30 01:17:56.466735 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458525 6221 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 01:17:56.466735 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458545 6221 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 01:17:56.466735 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458551 6221 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 01:17:56.466735 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458556 6221 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 01:17:56.466735 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458563 6221 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 01:17:56.466735 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458568 6221 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 01:17:56.466735 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458573 6221 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 01:17:56.466735 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458578 6221 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 01:17:56.466735 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458583 6221 flags.go:64] FLAG: --enable-server="true" Oct 30 01:17:56.466735 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458587 6221 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 01:17:56.466735 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458606 6221 flags.go:64] FLAG: --event-burst="100" Oct 30 01:17:56.466735 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458612 6221 flags.go:64] FLAG: --event-qps="50" Oct 30 01:17:56.466735 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458617 6221 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 01:17:56.466735 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458622 6221 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 01:17:56.466735 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458627 6221 flags.go:64] FLAG: --eviction-hard="" Oct 30 01:17:56.467894 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458633 6221 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 01:17:56.467894 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458638 6221 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 01:17:56.467894 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458643 6221 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 01:17:56.467894 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458651 6221 flags.go:64] FLAG: --eviction-soft="" Oct 30 01:17:56.467894 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458656 6221 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 01:17:56.467894 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458661 6221 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 01:17:56.467894 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458666 6221 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 01:17:56.467894 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458671 6221 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 01:17:56.467894 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458676 6221 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 01:17:56.467894 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458681 6221 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 01:17:56.467894 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458685 6221 flags.go:64] FLAG: --feature-gates="" Oct 30 01:17:56.467894 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458694 6221 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 01:17:56.467894 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458700 6221 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 01:17:56.467894 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458705 6221 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 01:17:56.467894 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458710 6221 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 01:17:56.467894 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458715 6221 flags.go:64] FLAG: --healthz-port="10248" Oct 30 01:17:56.467894 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458720 6221 flags.go:64] FLAG: --help="false" Oct 30 01:17:56.467894 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458725 6221 flags.go:64] FLAG: --hostname-override="ip-10-0-135-65.ec2.internal" Oct 30 01:17:56.467894 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458730 6221 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 01:17:56.467894 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458735 6221 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 01:17:56.467894 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458740 6221 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 01:17:56.467894 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458746 6221 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 01:17:56.467894 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458752 6221 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 01:17:56.468886 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458757 6221 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 01:17:56.468886 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458762 6221 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 01:17:56.468886 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458766 6221 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 01:17:56.468886 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458771 6221 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 01:17:56.468886 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458776 6221 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 01:17:56.468886 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458789 6221 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 01:17:56.468886 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458795 6221 flags.go:64] FLAG: --kube-reserved="" Oct 30 01:17:56.468886 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458800 6221 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 01:17:56.468886 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458805 6221 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 01:17:56.468886 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458810 6221 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 01:17:56.468886 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458815 6221 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 01:17:56.468886 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458820 6221 flags.go:64] FLAG: --lock-file="" Oct 30 01:17:56.468886 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458827 6221 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 01:17:56.468886 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458832 6221 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 01:17:56.468886 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458837 6221 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 01:17:56.468886 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458846 6221 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 01:17:56.468886 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458851 6221 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 01:17:56.468886 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458856 6221 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 01:17:56.468886 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458860 6221 flags.go:64] FLAG: --logging-format="text" Oct 30 01:17:56.468886 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458865 6221 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 01:17:56.468886 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458873 6221 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 01:17:56.468886 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458878 6221 flags.go:64] FLAG: --manifest-url="" Oct 30 01:17:56.468886 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458883 6221 flags.go:64] FLAG: --manifest-url-header="" Oct 30 01:17:56.468886 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458890 6221 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 01:17:56.468886 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458895 6221 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 01:17:56.469991 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458902 6221 flags.go:64] FLAG: --max-pods="110" Oct 30 01:17:56.469991 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458906 6221 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 01:17:56.469991 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458912 6221 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 01:17:56.469991 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458917 6221 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 01:17:56.469991 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458922 6221 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 01:17:56.469991 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458927 6221 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 01:17:56.469991 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458931 6221 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 01:17:56.469991 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458937 6221 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 01:17:56.469991 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458947 6221 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 01:17:56.469991 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458952 6221 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 01:17:56.469991 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458958 6221 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 01:17:56.469991 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458963 6221 flags.go:64] FLAG: --pod-cidr="" Oct 30 01:17:56.469991 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458968 6221 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 01:17:56.469991 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458981 6221 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 01:17:56.469991 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458986 6221 flags.go:64] FLAG: --pods-per-core="0" Oct 30 01:17:56.469991 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458991 6221 flags.go:64] FLAG: --port="10250" Oct 30 01:17:56.469991 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.458995 6221 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 01:17:56.469991 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.459001 6221 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00c04657cfc51c167" Oct 30 01:17:56.469991 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.459006 6221 flags.go:64] FLAG: --qos-reserved="" Oct 30 01:17:56.469991 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.459013 6221 flags.go:64] FLAG: --read-only-port="10255" Oct 30 01:17:56.469991 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.459018 6221 flags.go:64] FLAG: --register-node="true" Oct 30 01:17:56.469991 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.459023 6221 flags.go:64] FLAG: --register-with-taints="" Oct 30 01:17:56.469991 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.459029 6221 flags.go:64] FLAG: --registry-burst="10" Oct 30 01:17:56.469991 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.459034 6221 flags.go:64] FLAG: --registry-qps="5" Oct 30 01:17:56.469991 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.459038 6221 flags.go:64] FLAG: --reserved-cpus="" Oct 30 01:17:56.469991 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.459044 6221 flags.go:64] FLAG: --reserved-memory="" Oct 30 01:17:56.471110 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.459050 6221 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 01:17:56.471110 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.459058 6221 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 01:17:56.471110 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.459063 6221 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 01:17:56.471110 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.459068 6221 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 01:17:56.471110 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.459072 6221 flags.go:64] FLAG: --runonce="false" Oct 30 01:17:56.471110 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.459077 6221 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 01:17:56.471110 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.459082 6221 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 01:17:56.471110 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.459087 6221 flags.go:64] FLAG: --seccomp-default="false" Oct 30 01:17:56.471110 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.459092 6221 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 01:17:56.471110 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.459097 6221 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 01:17:56.471110 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.459102 6221 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 01:17:56.471110 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.459107 6221 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 01:17:56.471110 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.459112 6221 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 01:17:56.471110 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.459117 6221 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 01:17:56.471110 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.459121 6221 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 01:17:56.471110 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.459126 6221 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 01:17:56.471110 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.459131 6221 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 01:17:56.471110 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.459136 6221 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 01:17:56.471110 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.459141 6221 flags.go:64] FLAG: --system-cgroups="" Oct 30 01:17:56.471110 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.459145 6221 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 01:17:56.471110 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.459162 6221 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 01:17:56.471110 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.459167 6221 flags.go:64] FLAG: --tls-cert-file="" Oct 30 01:17:56.471110 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.459172 6221 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 01:17:56.471110 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.459185 6221 flags.go:64] FLAG: --tls-min-version="" Oct 30 01:17:56.471110 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.459190 6221 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 01:17:56.472310 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.459197 6221 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 01:17:56.472310 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.459201 6221 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 01:17:56.472310 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.459207 6221 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 01:17:56.472310 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.459212 6221 flags.go:64] FLAG: --v="2" Oct 30 01:17:56.472310 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.459219 6221 flags.go:64] FLAG: --version="false" Oct 30 01:17:56.472310 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.459225 6221 flags.go:64] FLAG: --vmodule="" Oct 30 01:17:56.472310 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.459231 6221 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 01:17:56.472310 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.459236 6221 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 01:17:56.472310 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459514 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:17:56.472310 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459523 6221 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:17:56.472310 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459546 6221 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:17:56.472310 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459551 6221 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:17:56.472310 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459556 6221 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:17:56.472310 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459561 6221 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:17:56.472310 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459566 6221 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:17:56.472310 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459570 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:17:56.472310 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459574 6221 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:17:56.472310 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459579 6221 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:17:56.472310 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459583 6221 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:17:56.472310 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459588 6221 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:17:56.472310 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459592 6221 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:17:56.473139 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459597 6221 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:17:56.473139 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459601 6221 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:17:56.473139 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459605 6221 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:17:56.473139 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459610 6221 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:17:56.473139 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459614 6221 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:17:56.473139 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459619 6221 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:17:56.473139 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459624 6221 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:17:56.473139 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459637 6221 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:17:56.473139 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459642 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:17:56.473139 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459646 6221 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:17:56.473139 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459651 6221 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:17:56.473139 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459659 6221 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:17:56.473139 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459663 6221 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:17:56.473139 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459667 6221 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:17:56.473139 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459672 6221 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:17:56.473139 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459676 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:17:56.473139 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459681 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:17:56.473139 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459685 6221 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:17:56.473139 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459689 6221 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:17:56.473996 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459697 6221 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:17:56.473996 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459701 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:17:56.473996 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459705 6221 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:17:56.473996 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459710 6221 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:17:56.473996 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459714 6221 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:17:56.473996 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459718 6221 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:17:56.473996 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459723 6221 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:17:56.473996 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459727 6221 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:17:56.473996 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459731 6221 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:17:56.473996 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459735 6221 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:17:56.473996 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459740 6221 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:17:56.473996 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459744 6221 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:17:56.473996 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459749 6221 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:17:56.473996 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459753 6221 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:17:56.473996 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459758 6221 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:17:56.473996 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459762 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:17:56.473996 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459767 6221 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:17:56.473996 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459771 6221 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:17:56.473996 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459775 6221 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:17:56.473996 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459780 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:17:56.474547 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459784 6221 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:17:56.474547 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459797 6221 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:17:56.474547 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459802 6221 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:17:56.474547 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459806 6221 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:17:56.474547 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459812 6221 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:17:56.474547 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459816 6221 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:17:56.474547 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459820 6221 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:17:56.474547 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459825 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:17:56.474547 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459830 6221 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:17:56.474547 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459834 6221 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:17:56.474547 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459838 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:17:56.474547 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459843 6221 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:17:56.474547 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459849 6221 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:17:56.474547 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459854 6221 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:17:56.474547 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459858 6221 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:17:56.474547 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459863 6221 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:17:56.474547 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459867 6221 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:17:56.474547 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459872 6221 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:17:56.474547 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459876 6221 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:17:56.474547 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459881 6221 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:17:56.475378 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459885 6221 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:17:56.475378 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459897 6221 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:17:56.475378 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459902 6221 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:17:56.475378 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459906 6221 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:17:56.475378 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459910 6221 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:17:56.475378 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459915 6221 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:17:56.475378 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459919 6221 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:17:56.475378 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459924 6221 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:17:56.475378 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459928 6221 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:17:56.475378 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459932 6221 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:17:56.475378 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459937 6221 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:17:56.475378 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459941 6221 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:17:56.475378 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459946 6221 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:17:56.475378 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459950 6221 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:17:56.475378 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459963 6221 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:17:56.475378 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459968 6221 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:17:56.475378 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459974 6221 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:17:56.475378 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459979 6221 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:17:56.475378 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459983 6221 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:17:56.475378 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459988 6221 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:17:56.476335 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459992 6221 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:17:56.476335 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.459997 6221 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:17:56.476335 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460001 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:17:56.476335 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460005 6221 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:17:56.476335 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460012 6221 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:17:56.476335 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460016 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:17:56.476335 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460021 6221 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:17:56.476335 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460025 6221 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:17:56.476335 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460030 6221 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:17:56.476335 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460036 6221 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:17:56.476335 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460043 6221 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:17:56.476335 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460048 6221 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:17:56.476335 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460053 6221 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:17:56.476335 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460058 6221 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:17:56.476335 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.460100 6221 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 01:17:56.476885 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460165 6221 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:17:56.476885 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460173 6221 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:17:56.476885 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460177 6221 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:17:56.476885 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460182 6221 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:17:56.476885 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460186 6221 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:17:56.476885 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460190 6221 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:17:56.476885 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460195 6221 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:17:56.476885 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460199 6221 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:17:56.476885 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460204 6221 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:17:56.476885 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460208 6221 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:17:56.476885 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460213 6221 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:17:56.476885 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460217 6221 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:17:56.476885 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460232 6221 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:17:56.476885 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460236 6221 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:17:56.476885 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460243 6221 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:17:56.476885 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460247 6221 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:17:56.476885 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460252 6221 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:17:56.476885 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460256 6221 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:17:56.476885 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460261 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:17:56.477450 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460265 6221 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:17:56.477450 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460269 6221 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:17:56.477450 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460276 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:17:56.477450 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460281 6221 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:17:56.477450 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460286 6221 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:17:56.477450 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460290 6221 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:17:56.477450 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460297 6221 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:17:56.477450 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460303 6221 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:17:56.477450 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460307 6221 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:17:56.477450 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460312 6221 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:17:56.477450 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460316 6221 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:17:56.477450 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460321 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:17:56.477450 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460327 6221 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:17:56.477450 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460333 6221 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:17:56.477450 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460338 6221 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:17:56.477450 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460343 6221 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:17:56.477450 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460348 6221 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:17:56.477450 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460352 6221 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:17:56.478006 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460356 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:17:56.478006 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460361 6221 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:17:56.478006 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460365 6221 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:17:56.478006 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460370 6221 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:17:56.478006 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460374 6221 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:17:56.478006 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460379 6221 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:17:56.478006 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460383 6221 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:17:56.478006 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460387 6221 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:17:56.478006 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460399 6221 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:17:56.478006 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460406 6221 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:17:56.478006 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460411 6221 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:17:56.478006 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460416 6221 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:17:56.478006 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460421 6221 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:17:56.478006 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460426 6221 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:17:56.478006 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460430 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:17:56.478006 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460434 6221 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:17:56.478006 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460441 6221 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:17:56.478006 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460446 6221 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:17:56.478006 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460450 6221 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:17:56.478006 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460455 6221 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:17:56.478564 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460459 6221 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:17:56.478564 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460464 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:17:56.478564 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460468 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:17:56.478564 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460473 6221 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:17:56.478564 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460477 6221 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:17:56.478564 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460481 6221 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:17:56.478564 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460490 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:17:56.478564 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460495 6221 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:17:56.478564 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460499 6221 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:17:56.478564 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460504 6221 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:17:56.478564 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460508 6221 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:17:56.478564 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460512 6221 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:17:56.478564 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460516 6221 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:17:56.478564 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460521 6221 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:17:56.478564 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460525 6221 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:17:56.478564 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460546 6221 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:17:56.478564 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460551 6221 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:17:56.478564 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460555 6221 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:17:56.478564 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460559 6221 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:17:56.478564 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460563 6221 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:17:56.479092 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460567 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:17:56.479092 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460582 6221 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:17:56.479092 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460587 6221 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:17:56.479092 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460591 6221 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:17:56.479092 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460596 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:17:56.479092 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460600 6221 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:17:56.479092 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460605 6221 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:17:56.479092 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460609 6221 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:17:56.479092 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460615 6221 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:17:56.479092 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460620 6221 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:17:56.479092 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460624 6221 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:17:56.479092 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460628 6221 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:17:56.479092 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460633 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:17:56.479092 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460638 6221 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:17:56.479092 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460642 6221 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:17:56.479092 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460646 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:17:56.479092 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460651 6221 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:17:56.479092 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460655 6221 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:17:56.479092 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460662 6221 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:17:56.479092 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460666 6221 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:17:56.479643 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460671 6221 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:17:56.479643 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460675 6221 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:17:56.479643 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460680 6221 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:17:56.479643 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460684 6221 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:17:56.479643 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460689 6221 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:17:56.479643 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460693 6221 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:17:56.479643 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460698 6221 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:17:56.479643 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460702 6221 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:17:56.479643 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.460706 6221 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:17:56.479643 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.465135 6221 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 01:17:56.479643 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.465153 6221 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 01:17:56.479643 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.465172 6221 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 01:17:56.479643 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.465176 6221 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 01:17:56.479643 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.465716 6221 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:17:56.479643 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.465743 6221 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:17:56.479643 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.465750 6221 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:17:56.479643 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.465764 6221 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:17:56.479643 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.465771 6221 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:17:56.479643 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.465776 6221 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:17:56.480179 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.465780 6221 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:17:56.480179 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.465786 6221 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:17:56.480179 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.465818 6221 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:17:56.480179 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.465824 6221 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:17:56.480179 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.465830 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:17:56.480179 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.465835 6221 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:17:56.480179 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.465839 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:17:56.480179 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.465844 6221 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:17:56.480179 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.465849 6221 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:17:56.480179 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.465860 6221 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:17:56.480179 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.465864 6221 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:17:56.480179 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.465868 6221 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:17:56.480179 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.465873 6221 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:17:56.480179 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.465878 6221 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:17:56.480179 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.465883 6221 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:17:56.480179 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.465888 6221 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:17:56.480179 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.465893 6221 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:17:56.480179 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.465897 6221 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:17:56.480179 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.465902 6221 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:17:56.480712 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.465906 6221 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:17:56.480712 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.465910 6221 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:17:56.480712 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.465919 6221 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:17:56.480712 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.465923 6221 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:17:56.480712 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.465928 6221 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:17:56.480712 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.465933 6221 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:17:56.480712 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.465938 6221 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:17:56.480712 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.465942 6221 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:17:56.480712 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.465946 6221 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:17:56.480712 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.465951 6221 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:17:56.480712 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.465955 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:17:56.480712 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.465959 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:17:56.480712 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.465964 6221 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:17:56.480712 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.465968 6221 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:17:56.480712 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.465972 6221 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:17:56.480712 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.465981 6221 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:17:56.480712 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466012 6221 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:17:56.480712 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466018 6221 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:17:56.480712 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466022 6221 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:17:56.481225 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466027 6221 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:17:56.481225 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466032 6221 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:17:56.481225 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466036 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:17:56.481225 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466040 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:17:56.481225 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466045 6221 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:17:56.481225 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466050 6221 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:17:56.481225 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466054 6221 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:17:56.481225 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466059 6221 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:17:56.481225 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466064 6221 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:17:56.481225 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466074 6221 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:17:56.481225 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466078 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:17:56.481225 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466083 6221 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:17:56.481225 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466088 6221 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:17:56.481225 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466092 6221 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:17:56.481225 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466097 6221 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:17:56.481225 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466102 6221 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:17:56.481225 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466106 6221 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:17:56.481225 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466111 6221 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:17:56.481225 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466116 6221 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:17:56.481225 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466121 6221 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:17:56.481849 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466125 6221 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:17:56.481849 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466135 6221 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:17:56.481849 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466139 6221 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:17:56.481849 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466144 6221 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:17:56.481849 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466149 6221 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:17:56.481849 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466153 6221 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:17:56.481849 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466158 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:17:56.481849 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466161 6221 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:17:56.481849 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466166 6221 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:17:56.481849 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466170 6221 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:17:56.481849 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466199 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:17:56.481849 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466204 6221 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:17:56.481849 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466209 6221 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:17:56.481849 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466213 6221 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:17:56.481849 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466222 6221 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:17:56.481849 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466226 6221 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:17:56.481849 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466231 6221 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:17:56.481849 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466235 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:17:56.481849 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466240 6221 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:17:56.482326 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466245 6221 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:17:56.482326 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466249 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:17:56.482326 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466254 6221 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:17:56.482326 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466259 6221 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:17:56.482326 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466264 6221 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:17:56.482326 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466268 6221 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:17:56.482326 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466272 6221 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:17:56.482326 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466277 6221 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:17:56.482326 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466286 6221 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:17:56.482326 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466290 6221 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:17:56.482326 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466295 6221 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:17:56.482326 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466299 6221 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:17:56.482326 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466303 6221 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:17:56.482326 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466308 6221 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:17:56.482326 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466312 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:17:56.482326 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466316 6221 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:17:56.482326 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466320 6221 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:17:56.482326 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466324 6221 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:17:56.482326 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466329 6221 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:17:56.482326 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466333 6221 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:17:56.482865 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466375 6221 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:17:56.482865 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466380 6221 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:17:56.482865 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466383 6221 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:17:56.482865 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.466598 6221 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 01:17:56.482865 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466872 6221 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:17:56.482865 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466884 6221 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:17:56.482865 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466890 6221 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:17:56.482865 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466896 6221 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:17:56.482865 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466901 6221 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:17:56.482865 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466907 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:17:56.482865 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466911 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:17:56.482865 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466916 6221 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:17:56.482865 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466921 6221 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:17:56.482865 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466925 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:17:56.482865 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466930 6221 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:17:56.482865 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466934 6221 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:17:56.483265 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466939 6221 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:17:56.483265 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466943 6221 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:17:56.483265 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466948 6221 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:17:56.483265 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466953 6221 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:17:56.483265 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466957 6221 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:17:56.483265 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466962 6221 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:17:56.483265 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466966 6221 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:17:56.483265 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466970 6221 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:17:56.483265 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466975 6221 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:17:56.483265 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466979 6221 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:17:56.483265 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466983 6221 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:17:56.483265 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466988 6221 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:17:56.483265 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466992 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:17:56.483265 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.466997 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:17:56.483265 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467001 6221 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:17:56.483265 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467006 6221 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:17:56.483265 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467010 6221 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:17:56.483265 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467015 6221 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:17:56.483265 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467019 6221 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:17:56.483743 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467023 6221 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:17:56.483743 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467028 6221 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:17:56.483743 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467032 6221 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:17:56.483743 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467037 6221 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:17:56.483743 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467041 6221 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:17:56.483743 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467046 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:17:56.483743 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467050 6221 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:17:56.483743 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467054 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:17:56.483743 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467059 6221 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:17:56.483743 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467063 6221 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:17:56.483743 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467067 6221 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:17:56.483743 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467074 6221 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:17:56.483743 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467079 6221 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:17:56.483743 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467084 6221 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:17:56.483743 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467088 6221 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:17:56.483743 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467093 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:17:56.483743 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467097 6221 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:17:56.483743 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467102 6221 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:17:56.483743 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467106 6221 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:17:56.484201 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467111 6221 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:17:56.484201 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467116 6221 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:17:56.484201 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467120 6221 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:17:56.484201 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467125 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:17:56.484201 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467129 6221 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:17:56.484201 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467134 6221 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:17:56.484201 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467138 6221 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:17:56.484201 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467142 6221 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:17:56.484201 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467147 6221 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:17:56.484201 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467151 6221 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:17:56.484201 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467155 6221 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:17:56.484201 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467160 6221 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:17:56.484201 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467164 6221 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:17:56.484201 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467169 6221 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:17:56.484201 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467174 6221 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:17:56.484201 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467179 6221 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:17:56.484201 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467183 6221 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:17:56.484201 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467187 6221 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:17:56.484201 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467192 6221 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:17:56.484201 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467198 6221 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:17:56.484736 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467204 6221 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:17:56.484736 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467209 6221 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:17:56.484736 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467214 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:17:56.484736 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467218 6221 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:17:56.484736 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467223 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:17:56.484736 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467235 6221 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:17:56.484736 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467240 6221 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:17:56.484736 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467244 6221 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:17:56.484736 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467248 6221 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:17:56.484736 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467253 6221 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:17:56.484736 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467257 6221 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:17:56.484736 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467262 6221 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:17:56.484736 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467267 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:17:56.484736 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467272 6221 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:17:56.484736 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467278 6221 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:17:56.484736 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467284 6221 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:17:56.484736 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467288 6221 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:17:56.484736 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467293 6221 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:17:56.484736 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467297 6221 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:17:56.484736 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467302 6221 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:17:56.485322 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467306 6221 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:17:56.485322 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467311 6221 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:17:56.485322 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467316 6221 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:17:56.485322 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467320 6221 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:17:56.485322 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467324 6221 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:17:56.485322 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467331 6221 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:17:56.485322 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467337 6221 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:17:56.485322 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467342 6221 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:17:56.485322 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467347 6221 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:17:56.485322 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467351 6221 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:17:56.485322 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467355 6221 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:17:56.485322 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467360 6221 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:17:56.485322 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467365 6221 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:17:56.485322 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467369 6221 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:17:56.485322 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467373 6221 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:17:56.485322 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467378 6221 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:17:56.485856 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.467422 6221 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 01:17:56.485856 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467474 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 01:17:56.485856 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467483 6221 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 01:17:56.485856 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467488 6221 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 01:17:56.485856 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467493 6221 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 01:17:56.485856 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467498 6221 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 01:17:56.485856 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467502 6221 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 01:17:56.485856 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467507 6221 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 01:17:56.485856 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467512 6221 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 01:17:56.485856 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467516 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 01:17:56.485856 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467521 6221 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 01:17:56.485856 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467525 6221 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 01:17:56.485856 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467546 6221 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 01:17:56.485856 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467550 6221 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 01:17:56.485856 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467555 6221 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 01:17:56.485856 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467560 6221 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 01:17:56.486358 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467564 6221 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 01:17:56.486358 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467569 6221 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 01:17:56.486358 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467573 6221 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 01:17:56.486358 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467577 6221 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 01:17:56.486358 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467581 6221 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 01:17:56.486358 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467586 6221 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 01:17:56.486358 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467592 6221 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 01:17:56.486358 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467599 6221 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 01:17:56.486358 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467604 6221 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 01:17:56.486358 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467610 6221 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 01:17:56.486358 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467615 6221 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 01:17:56.486358 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467619 6221 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 01:17:56.486358 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467623 6221 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 01:17:56.486358 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467627 6221 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 01:17:56.486358 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467631 6221 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 01:17:56.486358 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467636 6221 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 01:17:56.486358 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467640 6221 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 01:17:56.486358 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467645 6221 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 01:17:56.486358 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467649 6221 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 01:17:56.487048 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467655 6221 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 01:17:56.487048 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467660 6221 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 01:17:56.487048 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467664 6221 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 01:17:56.487048 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467669 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 01:17:56.487048 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467673 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 01:17:56.487048 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467677 6221 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 01:17:56.487048 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467682 6221 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 01:17:56.487048 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467687 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 01:17:56.487048 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467691 6221 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 01:17:56.487048 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467701 6221 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 01:17:56.487048 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467705 6221 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 01:17:56.487048 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467710 6221 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 01:17:56.487048 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467714 6221 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 01:17:56.487048 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467718 6221 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 01:17:56.487048 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467723 6221 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 01:17:56.487048 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467728 6221 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 01:17:56.487048 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467732 6221 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 01:17:56.487048 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467736 6221 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 01:17:56.487048 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467741 6221 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 01:17:56.487048 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467745 6221 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 01:17:56.487691 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467749 6221 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 01:17:56.487691 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467754 6221 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 01:17:56.487691 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467758 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 01:17:56.487691 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467762 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 01:17:56.487691 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467766 6221 feature_gate.go:386] unrecognized feature gate: Example Oct 30 01:17:56.487691 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467771 6221 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 01:17:56.487691 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467775 6221 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 01:17:56.487691 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467780 6221 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 01:17:56.487691 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467784 6221 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 01:17:56.487691 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467789 6221 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 01:17:56.487691 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467793 6221 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 01:17:56.487691 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467797 6221 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 01:17:56.487691 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467802 6221 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 01:17:56.487691 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467809 6221 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 01:17:56.487691 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467814 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 01:17:56.487691 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467818 6221 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 01:17:56.487691 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467822 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 01:17:56.487691 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467826 6221 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 01:17:56.487691 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467831 6221 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 01:17:56.488163 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467835 6221 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 01:17:56.488163 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467840 6221 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 01:17:56.488163 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467846 6221 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 01:17:56.488163 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467851 6221 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 01:17:56.488163 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467856 6221 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 01:17:56.488163 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467861 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 01:17:56.488163 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467865 6221 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 01:17:56.488163 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467870 6221 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 01:17:56.488163 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467874 6221 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 01:17:56.488163 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467879 6221 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 01:17:56.488163 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467884 6221 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 01:17:56.488163 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467888 6221 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 01:17:56.488163 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467893 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 01:17:56.488163 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467897 6221 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 01:17:56.488163 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467904 6221 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 01:17:56.488163 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467908 6221 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 01:17:56.488163 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467912 6221 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 01:17:56.488163 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467916 6221 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 01:17:56.488163 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467920 6221 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 01:17:56.488163 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467924 6221 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 01:17:56.488965 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467928 6221 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 01:17:56.488965 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467933 6221 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 01:17:56.488965 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467937 6221 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 01:17:56.488965 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467942 6221 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 01:17:56.488965 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467946 6221 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 01:17:56.488965 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467951 6221 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 01:17:56.488965 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467955 6221 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 01:17:56.488965 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467960 6221 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 01:17:56.488965 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467968 6221 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 01:17:56.488965 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467974 6221 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 01:17:56.488965 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467981 6221 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 01:17:56.488965 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467986 6221 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 01:17:56.488965 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:17:56.467991 6221 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 01:17:56.488965 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.468081 6221 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 01:17:56.488965 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.469643 6221 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 01:17:56.488965 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.469718 6221 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 01:17:56.488965 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.469983 6221 server.go:1014] "Starting client certificate rotation" Oct 30 01:17:56.488965 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.470113 6221 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 01:17:56.489639 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.470159 6221 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 01:11:45 +0000 UTC" deadline="2028-05-16 13:47:50.234687464 +0000 UTC" Oct 30 01:17:56.489639 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.470194 6221 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13548h29m53.764495542s" Oct 30 01:17:56.489639 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.470758 6221 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 01:17:56.489639 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.472580 6221 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 01:17:56.489639 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.474202 6221 log.go:25] "Validated CRI v1 runtime API" Oct 30 01:17:56.489639 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.476572 6221 log.go:25] "Validated CRI v1 image API" Oct 30 01:17:56.489639 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.477279 6221 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 01:17:56.489639 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.479376 6221 fs.go:135] Filesystem UUIDs: map[3ccc9cf7-84c2-442d-8b45-3ea4bc2c9a64:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 aab3d7db-4c1f-4cea-b5d4-f0a784d68598:/dev/nvme0n1p3] Oct 30 01:17:56.489886 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.479392 6221 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/041c582efbcd1445b58cc13ef4091c174b65eb6004559fe5429f4cacc4235d4f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/041c582efbcd1445b58cc13ef4091c174b65eb6004559fe5429f4cacc4235d4f/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/223f329baedfd1629bc7e494bc60ac6c4b654bcbc118acaff1ad96b96e5fc52a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/223f329baedfd1629bc7e494bc60ac6c4b654bcbc118acaff1ad96b96e5fc52a/userdata/shm major:0 minor:312 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3458374121b40d6bc0a516989f33c9354e56a50e5c95d56a86ca6dc6b5d22ff6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3458374121b40d6bc0a516989f33c9354e56a50e5c95d56a86ca6dc6b5d22ff6/userdata/shm major:0 minor:382 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3a6ccb5dd4cd9de9186c36c98eca4a4c4604c4435b29e1e08b52832bff369331/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3a6ccb5dd4cd9de9186c36c98eca4a4c4604c4435b29e1e08b52832bff369331/userdata/shm major:0 minor:378 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3eac419a77c7d5175c139a45a770985e2db87117baf8a3ce67ea4083e3e39ac6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3eac419a77c7d5175c139a45a770985e2db87117baf8a3ce67ea4083e3e39ac6/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/415e62159ff51389db9ac1f6f638080802e928cef05b934bed5ae6c9b0e64520/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/415e62159ff51389db9ac1f6f638080802e928cef05b934bed5ae6c9b0e64520/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/47d43ed550365a099b9f7367bde105cb6ea4ca15b5aa598ca8465137f2650ea4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/47d43ed550365a099b9f7367bde105cb6ea4ca15b5aa598ca8465137f2650ea4/userdata/shm major:0 minor:260 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/59c2774dd37f74558ea7652843c6b6f576330ff6c1df4affcb6e60d147d5aa10/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/59c2774dd37f74558ea7652843c6b6f576330ff6c1df4affcb6e60d147d5aa10/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/65640885d7180bd4cb191fa8b6d250f689ccfe4f63fb732180ae40bb71f3a4af/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/65640885d7180bd4cb191fa8b6d250f689ccfe4f63fb732180ae40bb71f3a4af/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/66cfed6041737b855c39214ebc841b78bd2197bc4a3cbd61dfec11a91da2bde2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/66cfed6041737b855c39214ebc841b78bd2197bc4a3cbd61dfec11a91da2bde2/userdata/shm major:0 minor:298 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/70df9bdcacf94f607d6ba53601e4133c206acac378e1e5de5ca27634d718bc16/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/70df9bdcacf94f607d6ba53601e4133c206acac378e1e5de5ca27634d718bc16/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/77762bb9330192855d2831289a47030363071dc9353470c76bece8a57c121c01/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/77762bb9330192855d2831289a47030363071dc9353470c76bece8a57c121c01/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7e4a875aaf934ddc22726c19608ce2cbafc5a2da200c7f9949bd1d2c7d549a2d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7e4a875aaf934ddc22726c19608ce2cbafc5a2da200c7f9949bd1d2c7d549a2d/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/92b783d75cb547e19ac8d760f2932bea9da42a7227ad2efa48365245c4a84441/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/92b783d75cb547e19ac8d760f2932bea9da42a7227ad2efa48365245c4a84441/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a095b5dc32b2cf300fed1b5e01fe9e5bb6ede4318878b8166dfe02367469e34c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a095b5dc32b2cf300fed1b5e01fe9e5bb6ede4318878b8166dfe02367469e34c/userdata/shm major:0 minor:375 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/af72ee1cf15eaa41e0b953be6acb5ee16168b2df9406dd7cdb0f04a5e176ca02/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/af72ee1cf15eaa41e0b953be6acb5ee16168b2df9406dd7cdb0f04a5e176ca02/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b3c286849918fe011ae31026c2e3dcc3e0f10a61248956794ccb5750291206b1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b3c286849918fe011ae31026c2e3dcc3e0f10a61248956794ccb5750291206b1/userdata/shm major:0 minor:284 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b92a8a76a7ebb37f3fe475c4ebcaa5d33d0f373c92559c7428367ddf74abe492/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b92a8a76a7ebb37f3fe475c4ebcaa5d33d0f373c92559c7428367ddf74abe492/userdata/shm major:0 minor:329 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d56876b5d0a50691f4c01b9e84fa5f3de12509e574beb636d58a060e7cc7fe9f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d56876b5d0a50691f4c01b9e84fa5f3de12509e574beb636d58a060e7cc7fe9f/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d69953189d01f4541102d21c4694ddde99fac0560b1983dda7d29603bd02ae87/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d69953189d01f4541102d21c4694ddde99fac0560b1983dda7d29603bd02ae87/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e357503c9c89fb4c2823d4997882a3185c18d2abb7aa618b6b31f657e077742d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e357503c9c89fb4c2823d4997882a3185c18d2abb7aa618b6b31f657e077742d/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e44db52d977777a8d97795e0523bae26cc5c1d0767befa45718e4c5062f195b1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e44db52d977777a8d97795e0523bae26cc5c1d0767befa45718e4c5062f195b1/userdata/shm major:0 minor:380 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f09ddf2fe29802b739251bcbfadbda34b9b134348512fc28b9f66fe7e9e5b497/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f09ddf2fe29802b739251bcbfadbda34b9b134348512fc28b9f66fe7e9e5b497/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/06f5cece-5273-44d1-90ba-67bf5dc0395b/volumes/kubernetes.io~projected/kube-api-access-jd48f:{mountpoint:/var/lib/kubelet/pods/06f5cece-5273-44d1-90ba-67bf5dc0395b/volumes/kubernetes.io~projected/kube-api-access-jd48f major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/06f5cece-5273-44d1-90ba-67bf5dc0395b/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/06f5cece-5273-44d1-90ba-67bf5dc0395b/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/380432b5-86bc-4dd3-a846-d7875d621a36/volumes/kubernetes.io~projected/kube-api-access-wjqdw:{mountpoint:/var/lib/kubelet/pods/380432b5-86bc-4dd3-a846-d7875d621a36/volumes/kubernetes.io~projected/kube-api-access-wjqdw major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/380432b5-86bc-4dd3-a846-d7875d621a36/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/380432b5-86bc-4dd3-a846-d7875d621a36/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:326 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/380432b5-86bc-4dd3-a846-d7875d621a36/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/380432b5-86bc-4dd3-a846-d7875d621a36/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:321 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/380432b5-86bc-4dd3-a846-d7875d621a36/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/380432b5-86bc-4dd3-a846-d7875d621a36/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4214a36a-fa6a-4290-aadc-2336881f84ba/volumes/kubernetes.io~projected/kube-api-access-t5k2j:{mountpoint:/var/lib/kubelet/pods/4214a36a-fa6a-4290-aadc-2336881f84ba/volumes/kubernetes.io~projected/kube-api-access-t5k2j major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4214a36a-fa6a-4290-aadc-2336881f84ba/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/4214a36a-fa6a-4290-aadc-2336881f84ba/volumes/kubernetes.io~secret/metrics-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5618fce7-e2ca-4423-887a-eea73fdd9d01/volumes/kubernetes.io~projected/kube-api-access-n7pbf:{mountpoint:/var/lib/kubelet/pods/5618fce7-e2ca-4423-887a-eea73fdd9d01/volumes/kubernetes.io~projected/kube-api-access-n7pbf major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d834c2a-f008-492e-9aac-362fe7a73672/volumes/kubernetes.io~projected/kube-api-access-sd9jh:{mountpoint:/var/lib/kubelet/pods/7d834c2a-f008-492e-9aac-362fe7a73672/volumes/kubernetes.io~projected/kube-api-access-sd9jh major:0 minor:372 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d834c2a-f008-492e-9aac-362fe7a73672/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/7d834c2a-f008-492e-9aac-362fe7a73672/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:367 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d834c2a-f008-492e-9aac-362fe7a73672/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/7d834c2a-f008-492e-9aac-362fe7a73672/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:371 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/806dfce0-6207-4feb-bee4-ce73cfd4b552/volumes/kubernetes.io~projected/kube-api-access-bgt8r:{mountpoint:/var/lib/kubelet/pods/806dfce0-6207-4feb-bee4-ce73cfd4b552/volumes/kubernetes.io~projected/kube-api-access-bgt8r major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/806dfce0-6207-4feb-bee4-ce73cfd4b552/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/806dfce0-6207-4feb-bee4-ce73cfd4b552/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/806dfce0-6207-4feb-bee4-ce73cfd4b552/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/806dfce0-6207-4feb-bee4-ce73cfd4b552/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e7e744e-9806-48ca-9d02-94bb07752f36/volumes/kubernetes.io~projected/kube-api-access-zz5xc:{mountpoint:/var/lib/kubelet/pods/8e7e744e-9806-48ca-9d02-94bb07752f36/volumes/kubernetes.io~projected/kube-api-access-zz5xc major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e7e744e-9806-48ca-9d02-94bb07752f36/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/8e7e744e-9806-48ca-9d02-94bb07752f36/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8e7e744e-9806-48ca-9d02-94bb07752f36/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/8e7e744e-9806-48ca-9d02-94bb07752f36/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94864d90-1dd8-472e-a414-29c6b6a4f8f6/volumes/kubernetes.io~projected/kube-api-access-5rmpm:{mountpoint:/var/lib/kubelet/pods/94864d90-1dd8-472e-a414-29c6b6a4f8f6/volumes/kubernetes.io~projected/kube-api-access-5rmpm major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/94864d90-1dd8-472e-a414-29c6b6a4f8f6/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/94864d90-1dd8-472e-a414-29c6b6a4f8f6/volumes/kubernetes.io~secret/metrics-certs major:0 minor:373 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b098bdf-b302-4954-907b-eadab89054a0/volumes/kubernetes.io~projected/kube-api-access-5nb6l:{mountpoint:/var/lib/kubelet/pods/9b098bdf-b302-4954-907b-eadab89054a0/volumes/kubernetes.io~projected/kube-api-access-5nb6l major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b098bdf-b302-4954-907b-eadab89054a0/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/9b098bdf-b302-4954-907b-eadab89054a0/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b098bdf-b302-4954-907b-eadab89054a0/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/9b098bdf-b302-4954-907b-eadab89054a0/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2ec1edc-b911-4537-ac00-f708ef6c9062/volumes/kubernetes.io~projected/kube-api-access-8qbvg:{mountpoint:/var/lib/kubelet/pods/a2ec1edc-b911-4537-ac00-f708ef6c9062/volumes/kubernetes.io~projected/kube-api-access-8qbvg major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4365b4f-a9d0-41eb-9d89-26591d3e3ed5/volumes/kubernetes.io~projected/kube-api-access-f67fc:{mountpoint:/var/lib/kubelet/pods/a4365b4f-a9d0-41eb-9d89-26591d3e3ed5/volumes/kubernetes.io~projected/kube-api-access-f67fc major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b981f97a-f238-4ef6-bbc8-e66cdd90dc90/volumes/kubernetes.io~projected/kube-api-access-hzhgj:{mountpoint:/var/lib/kubelet/pods/b981f97a-f238-4ef6-bbc8-e66cdd90dc90/volumes/kubernetes.io~projected/kube-api-access-hzhgj major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b981f97a-f238-4ef6-bbc8-e66cdd90dc90/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/b981f97a-f238-4ef6-bbc8-e66cdd90dc90/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bda81d83-62da-4f40-b668-29e2b0e51f35/volumes/kubernetes.io~projected/kube-api-access-p5wrm:{mountpoint:/var/lib/kubelet/pods/bda81d83-62da-4f40-b668-29e2b0e51f35/volumes/kubernetes.io~projected/kube-api-access-p5wrm major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c819c74a-0f8d-4a5e-9e1b-830196f6b117/volumes/kubernetes.io~projected/kube-api-access-4dhns:{mountpoint:/var/lib/kubelet/pods/c819c74a-0f8d-4a5e-9e1b-830196f6b117/volumes/kubernetes.io~projected/kube-api-access-4dhns major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d205185b-fc45-4a76-9719-20b00107abf1/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/d205185b-fc45-4a76-9719-20b00107abf1/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d205185b-fc45-4a76-9719-20b00107abf1/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/d205185b-fc45-4a76-9719-20b00107abf1/volume-subpaths/web-config/alertmanager/9 major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d205185b-fc45-4a76-9719-20b00107abf1/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/d205185b-fc45-4a76-9719-20b00107abf1/volume-subpaths/web-config/config-reloader/6 major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d205185b-fc45-4a76-9719-20b00107abf1/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/d205185b-fc45-4a76-9719-20b00107abf1/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d205185b-fc45-4a76-9719-20b00107abf1/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/d205185b-fc45-4a76-9719-20b00107abf1/volumes/kubernetes.io~empty-dir/config-out major:0 minor:292 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d205185b-fc45-4a76-9719-20b00107abf1/volumes/kubernetes.io~projected/kube-api-access-mcd88:{mountpoint:/var/lib/kubelet/pods/d205185b-fc45-4a76-9719-20b00107abf1/volumes/kubernetes.io~projected/kube-api-access-mcd88 major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d205185b-fc45-4a76-9719-20b00107abf1/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/d205185b-fc45-4a76-9719-20b00107abf1/volumes/kubernetes.io~projected/tls-assets major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d205185b-fc45-4a76-9719-20b00107abf1/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/d205185b-fc45-4a76-9719-20b00107abf1/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d205185b-fc45-4a76-9719-20b00107abf1/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/d205185b-fc45-4a76-9719-20b00107abf1/volumes/kubernetes.io~secret/config-volume major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d205185b-fc45-4a76-9719-20b00107abf1/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/d205185b-fc45-4a76-9719-20b00107abf1/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d205185b-fc45-4a76-9719-20b00107abf1/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/d205185b-fc45-4a76-9719-20b00107abf1/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d205185b-fc45-4a76-9719-20b00107abf1/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/d205185b-fc45-4a76-9719-20b00107abf1/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d205185b-fc45-4a76-9719-20b00107abf1/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/d205185b-fc45-4a76-9719-20b00107abf1/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d205185b-fc45-4a76-9719-20b00107abf1/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/d205185b-fc45-4a76-9719-20b00107abf1/volumes/kubernetes.io~secret/web-config major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d332bbb4-c699-44e8-96ba-516d8ca03e95/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/d332bbb4-c699-44e8-96ba-516d8ca03e95/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d332bbb4-c699-44e8-96ba-516d8ca03e95/volumes/kubernetes.io~projected/kube-api-access-8w5t7:{mountpoint:/var/lib/kubelet/pods/d332bbb4-c699-44e8-96ba-516d8ca03e95/volumes/kubernetes.io~projected/kube-api-access-8w5t7 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d332bbb4-c699-44e8-96ba-516d8ca03e95/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/d332bbb4-c699-44e8-96ba-516d8ca03e95/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db13cab2-9b76-44b5-8577-9f8f0d3f292e/volumes/kubernetes.io~projected/kube-api-access-l7frp:{mountpoint:/var/lib/kubelet/pods/db13cab2-9b76-44b5-8577-9f8f0d3f292e/volumes/kubernetes.io~projected/kube-api-access-l7frp major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db13cab2-9b76-44b5-8577-9f8f0d3f292e/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/db13cab2-9b76-44b5-8577-9f8f0d3f292e/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/db13cab2-9b76-44b5-8577-9f8f0d3f292e/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/db13cab2-9b76-44b5-8577-9f8f0d3f292e/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e09d6f2e-7cd5-47c3-b18b-482726067e90/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/e09d6f2e-7cd5-47c3-b18b-482726067e90/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e09d6f2e-7cd5-47c3-b18b-482726067e90/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/e09d6f2e-7cd5-47c3-b18b-482726067e90/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e09d6f2e-7cd5-47c3-b18b-482726067e90/volumes/kubernetes.io~projected/kube-api-access-nmptt:{mountpoint:/var/lib/kubelet/pods/e09d6f2e-7cd5-47c3-b18b-482726067e90/volumes/kubernetes.io~projected/kube-api-access-nmptt major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e30ba693-6284-485e-95fd-ad639627aa8d/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/e30ba693-6284-485e-95fd-ad639627aa8d/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ecb9d3db-582f-41a0-945d-7bd7e4892ff8/volumes/kubernetes.io~projected/kube-api-access-kdvvv:{mountpoint:/var/lib/kubelet/pods/ecb9d3db-582f-41a0-945d-7bd7e4892ff8/volumes/kubernetes.io~projected/kube-api-access-kdvvv major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f10aab9c-d4e6-45b5-a703-146114a31329/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/f10aab9c-d4e6-45b5-a703-146114a31329/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:374 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4602ac2-652b-4026-9a0d-d72162bb3258/volumes/kubernetes.io~projected/kube-api-access-89k2s:{mountpoint:/var/lib/kubelet/pods/f4602ac2-652b-4026-9a0d-d72162bb3258/volumes/kubernetes.io~projected/kube-api-access-89k2s major:0 minor:377 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/3dcee0d60142025a184fa0695bfceeb6ad2d1407775b5b3b186103e85ff2e82f/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/e4927f914b8050cc075039af377c76911f3010004f7375a1d344588a81205488/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/53410750f44a6fd67d865a29cf0beb6d3fdaa7b661c69505662331be73b01032/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/bc4ca087926f7478b2ec1556231035062f43fac0f425214a9b1d38fec525796c/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/5c1b8a9fe03d1557f2ba3612f7ca6e6f1bb177308c98b6449e0294b942f79dc3/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/b345343152651ad07996c9046f4bf3f3c1770ad8bbe9590d9c3d6fded5040012/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/6f4b6db1739684221412c2e50fa579bd2411053b960acb6e3424a5b41d318136/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/ef0effe5f39a65bdf159308ba66482a546d857fea6afaa0f421bdd3ee1804959/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/d51b0228bc8530639a84cbc314667bc7ced7486f9f8b6cebbd2b773082d4a772/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/4973728eb6ee8a1991581e4c7982c538fe1804d24712b3359e41c9e98ac66664/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/fedc44411fd102e49b5d1ea917733318db10185b62636caabae463f4820eb168/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-145:{mountpoint:/var/lib/containers/storage/overlay/d11202e3673c94dfa1eff7403abace99d0bd73123980e820c6a48ccdffb71563/merged major:0 minor:145 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/f31c82f063c251b91efd77f647c1a271a7ae0ffcea04c2f7694ebf0125e1869d/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/d87bba5e6b09a062c1b4e7c4661ee07f35c137f168b6dc00209f4d1e37a49dcf/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/f7ee718f0497b3869a50128e1aa53d3382461f0b1b9c1f328a1a7f524fbb6048/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/69c6007b495d54b3ed3fdf5ad2b0c9c26f4c516a59ef367cff9432f4d3320a0a/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-169:{mountpoint:/var/lib/containers/storage/overlay/ab0be71e1ebed5bf87e9b5af8e6b3829c916ac8bee0c3e545d127ecd867b178d/merged major:0 minor:169 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/1f8be0ae1ab3467050c36af9e99ccd82697d2d33182ec194925eed90d552255b/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/7d7a3dd11afa47e6448fb87a11dea20b1e1991f4fbcf38c4239fef84dbb4f5b9/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/14d78297556d18f8c4d638b52ebac7fbeddae1dc020939be2d66345aefbc651a/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/8b8d0318a49e130fa84d3cf8ba600e3d9c3b7ab15652347381c80e2099739c33/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/40a3a84a47a29f4013644544b847b7b685535abd39cf19688896d9ae92809061/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/91403255a79e63c2f048546488cd3340a704a9dbc372b5504f45bc0ed9714aec/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/557e1fe929d3c3c46637dd0e86182b1e01cae10fdd9446f476a9cb7d8cc63436/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/644d44f4b40443cad1a4b1bcbf3b349253ab7a094fb7701a1f166344fb1c216b/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/d1393cdbdb7796915c689c2fd92fd8f9b5d6376bff8efe8051b909379457a354/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/456535ce8c20e054134e9c3c02196140a6fa2698cd8d13b33854bcc8ee486223/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/0c9cf833b381df0d1ef45fa326cf3aaf7cd42a1ca545828c93c9ae6e684f3439/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/949f7b602f910a111b64da07d20890e942ed2d192f48cc445eb61be3a831b09c/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/5c9ac5e913403d391e67457bfb189ae45fa4203b2e543ee167ffd53501a3b739/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/e81ffbc21092f9bcd2ec7e970393490afdbc634f769f7313616e4ce386d64a8a/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/912392ad61eaec9f4b36773873792b37b86459820132ddb572a682ae368dc2db/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/912c086d14b010f987bfa2b3534f5ebaf3e62ca9ac05def20ca41200cdf99182/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/d69a805555e26df921ba05d2aa613beed2fe02a610880601c5f7acf8b6483edb/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/3b2fc06861bae9f8665bd2a54924682edbc611c80909e61218061f58121a8c90/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/9663504c07d3704f5f126389b9952fe967ff6b900a661a8c800c6a581fdbb3ec/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/d6bd52b5747f0e1a5d402ab646df0b9b7082dc6bff4a2edc5f8ed82724a94b26/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/e57fd3f7de7fe8d7dce7bb521b45f3430c1821e197c9d4c04c54ec8f68cbf451/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/7bbcec748d6bb0e8e8cbb8cad4a0269c8320b05713ff36673458142d800ea78c/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/f41a11478660c410309616fa4e9500435a9984c4c6340cbb236fbf9c2b7b67c5/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-319:{mountpoint:/var/lib/containers/storage/overlay/d3548b0cd029782acb14a510d50a8a0025ce2b69ef4e7ab48f11ac94ea125524/merged major:0 minor:319 fsType:overlay blockSize:0} overlay_0-331:{mountpoint:/var/lib/containers/storage/overlay/fe0e752d241e0d2a501f87cc59106a9fc1fb8f992c5f8ea195894799603136e7/merged major:0 minor:331 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/b5bfb3b7d7ea9514a00eb30f744749cff749f23262c34320731e89ebb805e31c/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/16b6949ca48dea1c4784f18ef13056dc1ae78ea1b16f92f128458b7a0f275b9f/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-341:{mountpoint:/var/lib/containers/storage/overlay/5ac7c02a30131b15b0b5919a5896adb309178ea1007fddb7232c36071af0281d/merged major:0 minor:341 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/cd6e727807bbf14694a728fcf367ed3161c1a6445eee28cd2ebf0873b12b81f1/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-351:{mountpoint:/var/lib/containers/storage/overlay/d17af80e26e074016198be901317386354d6b81df3bc4b4f374339a4761d5f36/merged major:0 minor:351 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/5a3a006289fe9f45cc991651d9ef27559345850ed7988e0ade33886fccb2dedc/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-361:{mountpoint:/var/lib/containers/storage/overlay/2968ac8011e95a3091c97a1e8e18c2a3968996fa90c5b18cb57449dc41d11c9e/merged major:0 minor:361 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/30d03ae3488033fef9afdc807eff044957820d3b5bdb94ada21b380241224b1f/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-386:{mountpoint:/var/lib/containers/storage/overlay/3df933d2b923594dcc31b7368060c8f556e31a7a4ce7c364c9eeb10f70e5fa17/merged major:0 minor:386 fsType:overlay blockSize:0} overlay_0-388:{mountpoint:/var/lib/containers/storage/overlay/0f27eba6dd977afa44a5cc27e15b707d6d37fb975d935a3cda00c7bf74d65370/merged major:0 minor:388 fsType:overlay blockSize:0} overlay_0-394:{mountpoint:/var/lib/containers/storage/overlay/78e169eca22cb28ceea5ddadd4f06e8cace9660e86a9b55315f71cb87d975078/merged major:0 minor:394 fsType:overlay blockSize:0} overlay_0-396:{mountpoint:/var/lib/containers/storage/overlay/800bfd1ebb1b0d80e9d948ffab85f371b44e18ab0ca4f24b257261c4a9e55a7f/merged major:0 minor:396 fsType:overlay blockSize:0} overlay_0-398:{mountpoint:/var/lib/containers/storage/overlay/9fdb46d545bc823ae06f76d77b26fdb67ae66bcdcedb8787978f2fd8c4c17bd4/merged major:0 minor:398 fsType:overlay blockSize:0} overlay_0-400:{mountpoint:/var/lib/containers/storage/overlay/8a7aa29bf68a4fcde7d2e988326144084329d61079a77a546ceb9d7014d5cd7c/merged major:0 minor:400 fsType:overlay blockSize:0} overlay_0-406:{mountpoint:/var/lib/containers/storage/overlay/0ce16404917be27b203fd61f9b1caa0c518a862e0fb41d8c44cd09900aee6be7/merged major:0 minor:406 fsType:overlay blockSize:0} overlay_0-411:{mountpoint:/var/lib/containers/storage/overlay/9f3ea5be330d707068237d64116aca15fbdf49f7f279ab55eb5b312e72513f7e/merged major:0 minor:411 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/af1030d75b93ee1497696b3fda4f40cea369e4f49157751b118ec585ad4db962/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/dabed7c4643d9531900c50695a30525cb15185e329de099d2bed3e7cbd98ba6a/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/a4028bec1684db3539f214ddf79ff5de6591a89e136efb533e83869be5aa10f6/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/54211ccf77bfefa7e76541559e933c7128a250cab26b50dd3077f1563dbf4701/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/351e333331793f62c4d5e78d17cea659a1957c2b1d90e7fdbbc42fcc72bae059/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/b28888a99f7d2552221460d56cff0c0f27e2d5d2b417fe0bea465440e9ea6b62/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/ccbc3a0f10d9d2c4920ccd5276affb01fc6b857c0c86c698e58fcbcb76edde33/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/2c52de119db3f6d994fac61173b71e96bb19b5f01fae3247eacdfa584fd10138/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/df18cd579a3cdfaf294301fb313c09821bdadfc974c1d227a1980be477e39405/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/d86c3350bc3ed7fbfecabc401d9ede237c16db2838e94fac91c65a08b9321fa2/merged major:0 minor:98 fsType:overlay blockSize:0}] Oct 30 01:17:56.490149 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.488418 6221 manager.go:217] Machine: {Timestamp:2026-10-30 01:17:56.487710711 +0000 UTC m=+0.065998039 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3136782 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec247f29ced8496ed48af6e583013cf9 SystemUUID:ec247f29-ced8-496e-d48a-f6e583013cf9 BootID:9720e4d8-b2b4-4456-be40-c0ce7217ea1f Filesystems:[{Device:/var/lib/kubelet/pods/94864d90-1dd8-472e-a414-29c6b6a4f8f6/volumes/kubernetes.io~projected/kube-api-access-5rmpm DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7e4a875aaf934ddc22726c19608ce2cbafc5a2da200c7f9949bd1d2c7d549a2d/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/77762bb9330192855d2831289a47030363071dc9353470c76bece8a57c121c01/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b3c286849918fe011ae31026c2e3dcc3e0f10a61248956794ccb5750291206b1/userdata/shm DeviceMajor:0 DeviceMinor:284 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d205185b-fc45-4a76-9719-20b00107abf1/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:289 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/806dfce0-6207-4feb-bee4-ce73cfd4b552/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:278 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/af72ee1cf15eaa41e0b953be6acb5ee16168b2df9406dd7cdb0f04a5e176ca02/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d205185b-fc45-4a76-9719-20b00107abf1/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:293 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-351 DeviceMajor:0 DeviceMinor:351 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-411 DeviceMajor:0 DeviceMinor:411 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f09ddf2fe29802b739251bcbfadbda34b9b134348512fc28b9f66fe7e9e5b497/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/db13cab2-9b76-44b5-8577-9f8f0d3f292e/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:314 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d332bbb4-c699-44e8-96ba-516d8ca03e95/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/b981f97a-f238-4ef6-bbc8-e66cdd90dc90/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/806dfce0-6207-4feb-bee4-ce73cfd4b552/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:282 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d205185b-fc45-4a76-9719-20b00107abf1/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:295 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/223f329baedfd1629bc7e494bc60ac6c4b654bcbc118acaff1ad96b96e5fc52a/userdata/shm DeviceMajor:0 DeviceMinor:312 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d205185b-fc45-4a76-9719-20b00107abf1/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:289 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d205185b-fc45-4a76-9719-20b00107abf1/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:294 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-394 DeviceMajor:0 DeviceMinor:394 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/92b783d75cb547e19ac8d760f2932bea9da42a7227ad2efa48365245c4a84441/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d205185b-fc45-4a76-9719-20b00107abf1/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:293 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/380432b5-86bc-4dd3-a846-d7875d621a36/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:326 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a4365b4f-a9d0-41eb-9d89-26591d3e3ed5/volumes/kubernetes.io~projected/kube-api-access-f67fc DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9b098bdf-b302-4954-907b-eadab89054a0/volumes/kubernetes.io~projected/kube-api-access-5nb6l DeviceMajor:0 DeviceMinor:259 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d205185b-fc45-4a76-9719-20b00107abf1/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:296 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f4602ac2-652b-4026-9a0d-d72162bb3258/volumes/kubernetes.io~projected/kube-api-access-89k2s DeviceMajor:0 DeviceMinor:377 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ecb9d3db-582f-41a0-945d-7bd7e4892ff8/volumes/kubernetes.io~projected/kube-api-access-kdvvv DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/041c582efbcd1445b58cc13ef4091c174b65eb6004559fe5429f4cacc4235d4f/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/66cfed6041737b855c39214ebc841b78bd2197bc4a3cbd61dfec11a91da2bde2/userdata/shm DeviceMajor:0 DeviceMinor:298 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4214a36a-fa6a-4290-aadc-2336881f84ba/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d205185b-fc45-4a76-9719-20b00107abf1/volumes/kubernetes.io~projected/kube-api-access-mcd88 DeviceMajor:0 DeviceMinor:297 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d205185b-fc45-4a76-9719-20b00107abf1/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:289 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/e09d6f2e-7cd5-47c3-b18b-482726067e90/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/65640885d7180bd4cb191fa8b6d250f689ccfe4f63fb732180ae40bb71f3a4af/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3eac419a77c7d5175c139a45a770985e2db87117baf8a3ce67ea4083e3e39ac6/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/70df9bdcacf94f607d6ba53601e4133c206acac378e1e5de5ca27634d718bc16/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a2ec1edc-b911-4537-ac00-f708ef6c9062/volumes/kubernetes.io~projected/kube-api-access-8qbvg DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c819c74a-0f8d-4a5e-9e1b-830196f6b117/volumes/kubernetes.io~projected/kube-api-access-4dhns DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4214a36a-fa6a-4290-aadc-2336881f84ba/volumes/kubernetes.io~projected/kube-api-access-t5k2j DeviceMajor:0 DeviceMinor:203 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/06f5cece-5273-44d1-90ba-67bf5dc0395b/volumes/kubernetes.io~projected/kube-api-access-jd48f DeviceMajor:0 DeviceMinor:204 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9b098bdf-b302-4954-907b-eadab89054a0/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:258 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/47d43ed550365a099b9f7367bde105cb6ea4ca15b5aa598ca8465137f2650ea4/userdata/shm DeviceMajor:0 DeviceMinor:260 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-386 DeviceMajor:0 DeviceMinor:386 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bda81d83-62da-4f40-b668-29e2b0e51f35/volumes/kubernetes.io~projected/kube-api-access-p5wrm DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/806dfce0-6207-4feb-bee4-ce73cfd4b552/volumes/kubernetes.io~projected/kube-api-access-bgt8r DeviceMajor:0 DeviceMinor:283 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-319 DeviceMajor:0 DeviceMinor:319 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-388 DeviceMajor:0 DeviceMinor:388 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-169 DeviceMajor:0 DeviceMinor:169 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d56876b5d0a50691f4c01b9e84fa5f3de12509e574beb636d58a060e7cc7fe9f/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d205185b-fc45-4a76-9719-20b00107abf1/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:290 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/415e62159ff51389db9ac1f6f638080802e928cef05b934bed5ae6c9b0e64520/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/380432b5-86bc-4dd3-a846-d7875d621a36/volumes/kubernetes.io~projected/kube-api-access-wjqdw DeviceMajor:0 DeviceMinor:328 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-331 DeviceMajor:0 DeviceMinor:331 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-361 DeviceMajor:0 DeviceMinor:361 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b92a8a76a7ebb37f3fe475c4ebcaa5d33d0f373c92559c7428367ddf74abe492/userdata/shm DeviceMajor:0 DeviceMinor:329 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7d834c2a-f008-492e-9aac-362fe7a73672/volumes/kubernetes.io~projected/kube-api-access-sd9jh DeviceMajor:0 DeviceMinor:372 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/94864d90-1dd8-472e-a414-29c6b6a4f8f6/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:373 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-400 DeviceMajor:0 DeviceMinor:400 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/d332bbb4-c699-44e8-96ba-516d8ca03e95/volumes/kubernetes.io~projected/kube-api-access-8w5t7 DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e44db52d977777a8d97795e0523bae26cc5c1d0767befa45718e4c5062f195b1/userdata/shm DeviceMajor:0 DeviceMinor:380 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e09d6f2e-7cd5-47c3-b18b-482726067e90/volumes/kubernetes.io~projected/kube-api-access-nmptt DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d205185b-fc45-4a76-9719-20b00107abf1/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:291 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5618fce7-e2ca-4423-887a-eea73fdd9d01/volumes/kubernetes.io~projected/kube-api-access-n7pbf DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/06f5cece-5273-44d1-90ba-67bf5dc0395b/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b981f97a-f238-4ef6-bbc8-e66cdd90dc90/volumes/kubernetes.io~projected/kube-api-access-hzhgj DeviceMajor:0 DeviceMinor:202 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9b098bdf-b302-4954-907b-eadab89054a0/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:253 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e09d6f2e-7cd5-47c3-b18b-482726067e90/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/380432b5-86bc-4dd3-a846-d7875d621a36/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:321 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a095b5dc32b2cf300fed1b5e01fe9e5bb6ede4318878b8166dfe02367469e34c/userdata/shm DeviceMajor:0 DeviceMinor:375 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3458374121b40d6bc0a516989f33c9354e56a50e5c95d56a86ca6dc6b5d22ff6/userdata/shm DeviceMajor:0 DeviceMinor:382 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-398 DeviceMajor:0 DeviceMinor:398 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-406 DeviceMajor:0 DeviceMinor:406 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8e7e744e-9806-48ca-9d02-94bb07752f36/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:225 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7d834c2a-f008-492e-9aac-362fe7a73672/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:371 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/59c2774dd37f74558ea7652843c6b6f576330ff6c1df4affcb6e60d147d5aa10/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d205185b-fc45-4a76-9719-20b00107abf1/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:289 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f10aab9c-d4e6-45b5-a703-146114a31329/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:374 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3a6ccb5dd4cd9de9186c36c98eca4a4c4604c4435b29e1e08b52832bff369331/userdata/shm DeviceMajor:0 DeviceMinor:378 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e30ba693-6284-485e-95fd-ad639627aa8d/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d69953189d01f4541102d21c4694ddde99fac0560b1983dda7d29603bd02ae87/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/db13cab2-9b76-44b5-8577-9f8f0d3f292e/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:315 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d332bbb4-c699-44e8-96ba-516d8ca03e95/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e357503c9c89fb4c2823d4997882a3185c18d2abb7aa618b6b31f657e077742d/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/380432b5-86bc-4dd3-a846-d7875d621a36/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:327 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8e7e744e-9806-48ca-9d02-94bb07752f36/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:229 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d205185b-fc45-4a76-9719-20b00107abf1/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:288 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/db13cab2-9b76-44b5-8577-9f8f0d3f292e/volumes/kubernetes.io~projected/kube-api-access-l7frp DeviceMajor:0 DeviceMinor:317 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7d834c2a-f008-492e-9aac-362fe7a73672/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:367 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-145 DeviceMajor:0 DeviceMinor:145 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8e7e744e-9806-48ca-9d02-94bb07752f36/volumes/kubernetes.io~projected/kube-api-access-zz5xc DeviceMajor:0 DeviceMinor:230 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d205185b-fc45-4a76-9719-20b00107abf1/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:292 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-341 DeviceMajor:0 DeviceMinor:341 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-396 DeviceMajor:0 DeviceMinor:396 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:041c582efbcd144 MacAddress:5a:c2:22:45:0b:1e Speed:10000 Mtu:8901} {Name:223f329baedfd16 MacAddress:5a:c6:11:57:7d:93 Speed:10000 Mtu:8901} {Name:3458374121b40d6 MacAddress:2e:75:75:6c:5a:5c Speed:10000 Mtu:8901} {Name:3a6ccb5dd4cd9de MacAddress:76:f0:e4:b4:72:8e Speed:10000 Mtu:8901} {Name:47d43ed550365a0 MacAddress:02:e0:9e:a0:ee:6e Speed:10000 Mtu:8901} {Name:59c2774dd37f745 MacAddress:3a:8a:f7:70:df:8f Speed:10000 Mtu:8901} {Name:66cfed6041737b8 MacAddress:9e:9c:9c:8e:61:24 Speed:10000 Mtu:8901} {Name:70df9bdcacf94f6 MacAddress:4e:76:ff:58:8b:10 Speed:10000 Mtu:8901} {Name:a095b5dc32b2cf3 MacAddress:3e:b2:1a:d6:93:dd Speed:10000 Mtu:8901} {Name:b92a8a76a7ebb37 MacAddress:82:ba:c2:28:56:e1 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:c1:68:de:53 Speed:0 Mtu:9001} {Name:br-int MacAddress:4a:5e:62:04:7f:0a Speed:0 Mtu:8901} {Name:d56876b5d0a5069 MacAddress:a2:6d:c1:8e:81:11 Speed:10000 Mtu:8901} {Name:e44db52d977777a MacAddress:82:76:64:56:1c:3f Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:c1:68:de:53 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:0a:51:ad:4e:14:4a Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:46:27:7e:4f:14:bf Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 01:17:56.490349 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.490157 6221 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 01:17:56.490349 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.490217 6221 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 01:17:56.490549 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.490503 6221 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 01:17:56.490689 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.490552 6221 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-65.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 01:17:56.490737 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.490691 6221 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 01:17:56.490737 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.490700 6221 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 01:17:56.490737 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.490710 6221 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 01:17:56.490737 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.490730 6221 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 01:17:56.490873 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.490829 6221 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 01:17:56.491007 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.490996 6221 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 01:17:56.491131 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.491121 6221 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 01:17:56.491168 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.491159 6221 kubelet.go:493] "Attempting to sync node with API server" Oct 30 01:17:56.491197 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.491177 6221 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 01:17:56.491197 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.491188 6221 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 01:17:56.491250 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.491198 6221 kubelet.go:405] "Adding apiserver pod source" Oct 30 01:17:56.491250 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.491206 6221 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 01:17:56.492102 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.492086 6221 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 01:17:56.492150 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.492105 6221 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 01:17:56.492701 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.492683 6221 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 01:17:56.495252 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.495236 6221 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 01:17:56.495252 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.495250 6221 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 01:17:56.495357 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.495348 6221 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 01:17:56.495570 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.495557 6221 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 01:17:56.495603 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.495589 6221 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 01:17:56.495710 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.495699 6221 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 01:17:56.495739 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.495720 6221 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 01:17:56.495739 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.495731 6221 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 01:17:56.495739 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.495737 6221 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 01:17:56.495823 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.495743 6221 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 01:17:56.495823 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.495749 6221 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 01:17:56.495823 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.495755 6221 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 01:17:56.495823 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.495761 6221 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 01:17:56.495823 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.495769 6221 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 01:17:56.495823 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.495775 6221 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 01:17:56.495823 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.495782 6221 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 01:17:56.495823 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.495792 6221 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 01:17:56.496027 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.495827 6221 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 01:17:56.496027 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.495833 6221 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 01:17:56.499277 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.499264 6221 server.go:1263] "Started kubelet" Oct 30 01:17:56.499428 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.499334 6221 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 01:17:56.499760 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.499617 6221 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 01:17:56.499864 ip-10-0-135-65 systemd[1]: Started Kubernetes Kubelet. Oct 30 01:17:56.500023 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.499964 6221 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 01:17:56.500512 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.500378 6221 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 01:17:56.501912 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.501891 6221 server.go:317] "Adding debug handlers to kubelet server" Oct 30 01:17:56.508962 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.506839 6221 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 01:17:56.508962 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.508074 6221 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 01:17:56.509288 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.509232 6221 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 01:11:46 +0000 UTC" deadline="2028-05-31 17:47:49.620072106 +0000 UTC" Oct 30 01:17:56.509483 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.509287 6221 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13912h29m53.11078877s" Oct 30 01:17:56.509483 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.509426 6221 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 01:17:56.509483 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.509440 6221 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 01:17:56.509755 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.509736 6221 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 01:17:56.511308 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:17:56.510020 6221 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-135-65.ec2.internal\" not found" Oct 30 01:17:56.511308 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.510855 6221 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 01:17:56.511308 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.510883 6221 factory.go:55] Registering systemd factory Oct 30 01:17:56.511308 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.510925 6221 factory.go:223] Registration of the systemd container factory successfully Oct 30 01:17:56.512484 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.512463 6221 factory.go:153] Registering CRI-O factory Oct 30 01:17:56.512484 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.512483 6221 factory.go:223] Registration of the crio container factory successfully Oct 30 01:17:56.512646 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.512501 6221 factory.go:103] Registering Raw factory Oct 30 01:17:56.512646 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.512516 6221 manager.go:1199] Started watching for new ooms in manager Oct 30 01:17:56.512884 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.512867 6221 manager.go:319] Starting recovery of all containers Oct 30 01:17:56.514868 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.514827 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d205185b-fc45-4a76-9719-20b00107abf1" volumeName="kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 01:17:56.514970 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.514869 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db13cab2-9b76-44b5-8577-9f8f0d3f292e" volumeName="kubernetes.io/configmap/db13cab2-9b76-44b5-8577-9f8f0d3f292e-console-config" seLinuxMountContext="" Oct 30 01:17:56.514970 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.514886 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4602ac2-652b-4026-9a0d-d72162bb3258" volumeName="kubernetes.io/projected/f4602ac2-652b-4026-9a0d-d72162bb3258-kube-api-access-89k2s" seLinuxMountContext="" Oct 30 01:17:56.514970 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.514901 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="806dfce0-6207-4feb-bee4-ce73cfd4b552" volumeName="kubernetes.io/configmap/806dfce0-6207-4feb-bee4-ce73cfd4b552-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 01:17:56.514970 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.514915 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="380432b5-86bc-4dd3-a846-d7875d621a36" volumeName="kubernetes.io/configmap/380432b5-86bc-4dd3-a846-d7875d621a36-metrics-server-audit-profiles" seLinuxMountContext="" Oct 30 01:17:56.514970 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.514929 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5618fce7-e2ca-4423-887a-eea73fdd9d01" volumeName="kubernetes.io/projected/5618fce7-e2ca-4423-887a-eea73fdd9d01-kube-api-access-n7pbf" seLinuxMountContext="" Oct 30 01:17:56.514970 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.514941 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bda81d83-62da-4f40-b668-29e2b0e51f35" volumeName="kubernetes.io/projected/bda81d83-62da-4f40-b668-29e2b0e51f35-kube-api-access-p5wrm" seLinuxMountContext="" Oct 30 01:17:56.514970 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.514965 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d332bbb4-c699-44e8-96ba-516d8ca03e95" volumeName="kubernetes.io/projected/d332bbb4-c699-44e8-96ba-516d8ca03e95-kube-api-access-8w5t7" seLinuxMountContext="" Oct 30 01:17:56.515308 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.514982 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c819c74a-0f8d-4a5e-9e1b-830196f6b117" volumeName="kubernetes.io/projected/c819c74a-0f8d-4a5e-9e1b-830196f6b117-kube-api-access-4dhns" seLinuxMountContext="" Oct 30 01:17:56.515308 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.514995 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d834c2a-f008-492e-9aac-362fe7a73672" volumeName="kubernetes.io/configmap/7d834c2a-f008-492e-9aac-362fe7a73672-trusted-ca-bundle" seLinuxMountContext="" Oct 30 01:17:56.515308 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515008 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d834c2a-f008-492e-9aac-362fe7a73672" volumeName="kubernetes.io/projected/7d834c2a-f008-492e-9aac-362fe7a73672-kube-api-access-sd9jh" seLinuxMountContext="" Oct 30 01:17:56.515308 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515021 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d205185b-fc45-4a76-9719-20b00107abf1" volumeName="kubernetes.io/empty-dir/d205185b-fc45-4a76-9719-20b00107abf1-config-out" seLinuxMountContext="" Oct 30 01:17:56.515308 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515034 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d834c2a-f008-492e-9aac-362fe7a73672" volumeName="kubernetes.io/secret/7d834c2a-f008-492e-9aac-362fe7a73672-console-oauth-config" seLinuxMountContext="" Oct 30 01:17:56.515308 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515047 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e09d6f2e-7cd5-47c3-b18b-482726067e90" volumeName="kubernetes.io/empty-dir/e09d6f2e-7cd5-47c3-b18b-482726067e90-etc-tuned" seLinuxMountContext="" Oct 30 01:17:56.515308 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515075 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06f5cece-5273-44d1-90ba-67bf5dc0395b" volumeName="kubernetes.io/empty-dir/06f5cece-5273-44d1-90ba-67bf5dc0395b-data-volume" seLinuxMountContext="" Oct 30 01:17:56.515308 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515089 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4214a36a-fa6a-4290-aadc-2336881f84ba" volumeName="kubernetes.io/empty-dir/4214a36a-fa6a-4290-aadc-2336881f84ba-tmp-dir" seLinuxMountContext="" Oct 30 01:17:56.515308 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515102 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="806dfce0-6207-4feb-bee4-ce73cfd4b552" volumeName="kubernetes.io/projected/806dfce0-6207-4feb-bee4-ce73cfd4b552-kube-api-access-bgt8r" seLinuxMountContext="" Oct 30 01:17:56.515308 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515114 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e7e744e-9806-48ca-9d02-94bb07752f36" volumeName="kubernetes.io/configmap/8e7e744e-9806-48ca-9d02-94bb07752f36-console-config" seLinuxMountContext="" Oct 30 01:17:56.515308 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515126 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d205185b-fc45-4a76-9719-20b00107abf1" volumeName="kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-cluster-tls-config" seLinuxMountContext="" Oct 30 01:17:56.515308 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515138 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db13cab2-9b76-44b5-8577-9f8f0d3f292e" volumeName="kubernetes.io/configmap/db13cab2-9b76-44b5-8577-9f8f0d3f292e-service-ca" seLinuxMountContext="" Oct 30 01:17:56.515308 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515151 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db13cab2-9b76-44b5-8577-9f8f0d3f292e" volumeName="kubernetes.io/configmap/db13cab2-9b76-44b5-8577-9f8f0d3f292e-trusted-ca-bundle" seLinuxMountContext="" Oct 30 01:17:56.515308 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515164 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db13cab2-9b76-44b5-8577-9f8f0d3f292e" volumeName="kubernetes.io/empty-dir/db13cab2-9b76-44b5-8577-9f8f0d3f292e-tmp" seLinuxMountContext="" Oct 30 01:17:56.515308 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515187 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="806dfce0-6207-4feb-bee4-ce73cfd4b552" volumeName="kubernetes.io/configmap/806dfce0-6207-4feb-bee4-ce73cfd4b552-metrics-client-ca" seLinuxMountContext="" Oct 30 01:17:56.515308 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515211 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4214a36a-fa6a-4290-aadc-2336881f84ba" volumeName="kubernetes.io/projected/4214a36a-fa6a-4290-aadc-2336881f84ba-kube-api-access-t5k2j" seLinuxMountContext="" Oct 30 01:17:56.515308 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515224 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="380432b5-86bc-4dd3-a846-d7875d621a36" volumeName="kubernetes.io/secret/380432b5-86bc-4dd3-a846-d7875d621a36-secret-metrics-server-tls" seLinuxMountContext="" Oct 30 01:17:56.515308 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515237 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="806dfce0-6207-4feb-bee4-ce73cfd4b552" volumeName="kubernetes.io/secret/806dfce0-6207-4feb-bee4-ce73cfd4b552-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 01:17:56.515308 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515252 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94864d90-1dd8-472e-a414-29c6b6a4f8f6" volumeName="kubernetes.io/secret/94864d90-1dd8-472e-a414-29c6b6a4f8f6-metrics-certs" seLinuxMountContext="" Oct 30 01:17:56.515308 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515266 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b098bdf-b302-4954-907b-eadab89054a0" volumeName="kubernetes.io/projected/9b098bdf-b302-4954-907b-eadab89054a0-kube-api-access-5nb6l" seLinuxMountContext="" Oct 30 01:17:56.515308 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515289 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d332bbb4-c699-44e8-96ba-516d8ca03e95" volumeName="kubernetes.io/configmap/d332bbb4-c699-44e8-96ba-516d8ca03e95-env-overrides" seLinuxMountContext="" Oct 30 01:17:56.515308 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515301 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e30ba693-6284-485e-95fd-ad639627aa8d" volumeName="kubernetes.io/secret/e30ba693-6284-485e-95fd-ad639627aa8d-agent-certs" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515325 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bda81d83-62da-4f40-b668-29e2b0e51f35" volumeName="kubernetes.io/configmap/bda81d83-62da-4f40-b668-29e2b0e51f35-multus-daemon-config" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515353 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db13cab2-9b76-44b5-8577-9f8f0d3f292e" volumeName="kubernetes.io/secret/db13cab2-9b76-44b5-8577-9f8f0d3f292e-console-serving-cert" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515373 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ecb9d3db-582f-41a0-945d-7bd7e4892ff8" volumeName="kubernetes.io/configmap/ecb9d3db-582f-41a0-945d-7bd7e4892ff8-serviceca" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515387 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06f5cece-5273-44d1-90ba-67bf5dc0395b" volumeName="kubernetes.io/configmap/06f5cece-5273-44d1-90ba-67bf5dc0395b-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515402 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="380432b5-86bc-4dd3-a846-d7875d621a36" volumeName="kubernetes.io/projected/380432b5-86bc-4dd3-a846-d7875d621a36-kube-api-access-wjqdw" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515416 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e7e744e-9806-48ca-9d02-94bb07752f36" volumeName="kubernetes.io/secret/8e7e744e-9806-48ca-9d02-94bb07752f36-console-serving-cert" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515428 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d205185b-fc45-4a76-9719-20b00107abf1" volumeName="kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-web-config" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515441 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d332bbb4-c699-44e8-96ba-516d8ca03e95" volumeName="kubernetes.io/secret/d332bbb4-c699-44e8-96ba-516d8ca03e95-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515456 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e09d6f2e-7cd5-47c3-b18b-482726067e90" volumeName="kubernetes.io/empty-dir/e09d6f2e-7cd5-47c3-b18b-482726067e90-tmp" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515469 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e09d6f2e-7cd5-47c3-b18b-482726067e90" volumeName="kubernetes.io/projected/e09d6f2e-7cd5-47c3-b18b-482726067e90-kube-api-access-nmptt" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515482 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ecb9d3db-582f-41a0-945d-7bd7e4892ff8" volumeName="kubernetes.io/projected/ecb9d3db-582f-41a0-945d-7bd7e4892ff8-kube-api-access-kdvvv" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515494 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e7e744e-9806-48ca-9d02-94bb07752f36" volumeName="kubernetes.io/configmap/8e7e744e-9806-48ca-9d02-94bb07752f36-oauth-serving-cert" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515508 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e7e744e-9806-48ca-9d02-94bb07752f36" volumeName="kubernetes.io/empty-dir/8e7e744e-9806-48ca-9d02-94bb07752f36-tmp" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515520 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d205185b-fc45-4a76-9719-20b00107abf1" volumeName="kubernetes.io/projected/d205185b-fc45-4a76-9719-20b00107abf1-tls-assets" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515548 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="380432b5-86bc-4dd3-a846-d7875d621a36" volumeName="kubernetes.io/configmap/380432b5-86bc-4dd3-a846-d7875d621a36-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515560 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="380432b5-86bc-4dd3-a846-d7875d621a36" volumeName="kubernetes.io/secret/380432b5-86bc-4dd3-a846-d7875d621a36-secret-metrics-server-client-certs" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515574 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="806dfce0-6207-4feb-bee4-ce73cfd4b552" volumeName="kubernetes.io/secret/806dfce0-6207-4feb-bee4-ce73cfd4b552-node-exporter-tls" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515586 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b098bdf-b302-4954-907b-eadab89054a0" volumeName="kubernetes.io/secret/9b098bdf-b302-4954-907b-eadab89054a0-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515599 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b981f97a-f238-4ef6-bbc8-e66cdd90dc90" volumeName="kubernetes.io/secret/b981f97a-f238-4ef6-bbc8-e66cdd90dc90-cert" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515611 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d205185b-fc45-4a76-9719-20b00107abf1" volumeName="kubernetes.io/projected/d205185b-fc45-4a76-9719-20b00107abf1-kube-api-access-mcd88" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515624 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db13cab2-9b76-44b5-8577-9f8f0d3f292e" volumeName="kubernetes.io/projected/db13cab2-9b76-44b5-8577-9f8f0d3f292e-kube-api-access-l7frp" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515635 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c819c74a-0f8d-4a5e-9e1b-830196f6b117" volumeName="kubernetes.io/empty-dir/c819c74a-0f8d-4a5e-9e1b-830196f6b117-tmp-dir" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515651 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d834c2a-f008-492e-9aac-362fe7a73672" volumeName="kubernetes.io/configmap/7d834c2a-f008-492e-9aac-362fe7a73672-console-config" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515666 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d834c2a-f008-492e-9aac-362fe7a73672" volumeName="kubernetes.io/configmap/7d834c2a-f008-492e-9aac-362fe7a73672-service-ca" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515681 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d834c2a-f008-492e-9aac-362fe7a73672" volumeName="kubernetes.io/empty-dir/7d834c2a-f008-492e-9aac-362fe7a73672-tmp" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515696 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e7e744e-9806-48ca-9d02-94bb07752f36" volumeName="kubernetes.io/configmap/8e7e744e-9806-48ca-9d02-94bb07752f36-service-ca" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515741 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="94864d90-1dd8-472e-a414-29c6b6a4f8f6" volumeName="kubernetes.io/projected/94864d90-1dd8-472e-a414-29c6b6a4f8f6-kube-api-access-5rmpm" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515754 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2ec1edc-b911-4537-ac00-f708ef6c9062" volumeName="kubernetes.io/configmap/a2ec1edc-b911-4537-ac00-f708ef6c9062-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515767 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4365b4f-a9d0-41eb-9d89-26591d3e3ed5" volumeName="kubernetes.io/projected/a4365b4f-a9d0-41eb-9d89-26591d3e3ed5-kube-api-access-f67fc" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515791 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="380432b5-86bc-4dd3-a846-d7875d621a36" volumeName="kubernetes.io/secret/380432b5-86bc-4dd3-a846-d7875d621a36-client-ca-bundle" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515804 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06f5cece-5273-44d1-90ba-67bf5dc0395b" volumeName="kubernetes.io/secret/06f5cece-5273-44d1-90ba-67bf5dc0395b-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515825 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e7e744e-9806-48ca-9d02-94bb07752f36" volumeName="kubernetes.io/projected/8e7e744e-9806-48ca-9d02-94bb07752f36-kube-api-access-zz5xc" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515838 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8e7e744e-9806-48ca-9d02-94bb07752f36" volumeName="kubernetes.io/secret/8e7e744e-9806-48ca-9d02-94bb07752f36-console-oauth-config" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515851 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b098bdf-b302-4954-907b-eadab89054a0" volumeName="kubernetes.io/secret/9b098bdf-b302-4954-907b-eadab89054a0-prometheus-operator-tls" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515863 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2ec1edc-b911-4537-ac00-f708ef6c9062" volumeName="kubernetes.io/configmap/a2ec1edc-b911-4537-ac00-f708ef6c9062-cni-binary-copy" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515875 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bda81d83-62da-4f40-b668-29e2b0e51f35" volumeName="kubernetes.io/configmap/bda81d83-62da-4f40-b668-29e2b0e51f35-cni-binary-copy" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515886 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d205185b-fc45-4a76-9719-20b00107abf1" volumeName="kubernetes.io/configmap/d205185b-fc45-4a76-9719-20b00107abf1-metrics-client-ca" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515902 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b098bdf-b302-4954-907b-eadab89054a0" volumeName="kubernetes.io/configmap/9b098bdf-b302-4954-907b-eadab89054a0-metrics-client-ca" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515916 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2ec1edc-b911-4537-ac00-f708ef6c9062" volumeName="kubernetes.io/projected/a2ec1edc-b911-4537-ac00-f708ef6c9062-kube-api-access-8qbvg" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515959 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d205185b-fc45-4a76-9719-20b00107abf1" volumeName="kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515974 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d205185b-fc45-4a76-9719-20b00107abf1" volumeName="kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-config-volume" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.515989 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e30ba693-6284-485e-95fd-ad639627aa8d" volumeName="kubernetes.io/configmap/e30ba693-6284-485e-95fd-ad639627aa8d-konnectivity-ca" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.516005 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f10aab9c-d4e6-45b5-a703-146114a31329" volumeName="kubernetes.io/secret/f10aab9c-d4e6-45b5-a703-146114a31329-original-pull-secret" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.516023 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06f5cece-5273-44d1-90ba-67bf5dc0395b" volumeName="kubernetes.io/projected/06f5cece-5273-44d1-90ba-67bf5dc0395b-kube-api-access-jd48f" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.516040 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d834c2a-f008-492e-9aac-362fe7a73672" volumeName="kubernetes.io/configmap/7d834c2a-f008-492e-9aac-362fe7a73672-oauth-serving-cert" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.516054 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d205185b-fc45-4a76-9719-20b00107abf1" volumeName="kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.516064 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d332bbb4-c699-44e8-96ba-516d8ca03e95" volumeName="kubernetes.io/configmap/d332bbb4-c699-44e8-96ba-516d8ca03e95-ovnkube-script-lib" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.516087 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db13cab2-9b76-44b5-8577-9f8f0d3f292e" volumeName="kubernetes.io/configmap/db13cab2-9b76-44b5-8577-9f8f0d3f292e-oauth-serving-cert" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.516101 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="db13cab2-9b76-44b5-8577-9f8f0d3f292e" volumeName="kubernetes.io/secret/db13cab2-9b76-44b5-8577-9f8f0d3f292e-console-oauth-config" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.516124 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="380432b5-86bc-4dd3-a846-d7875d621a36" volumeName="kubernetes.io/empty-dir/380432b5-86bc-4dd3-a846-d7875d621a36-audit-log" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.516140 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d205185b-fc45-4a76-9719-20b00107abf1" volumeName="kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.516154 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4214a36a-fa6a-4290-aadc-2336881f84ba" volumeName="kubernetes.io/configmap/4214a36a-fa6a-4290-aadc-2336881f84ba-config-volume" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.516167 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5618fce7-e2ca-4423-887a-eea73fdd9d01" volumeName="kubernetes.io/configmap/5618fce7-e2ca-4423-887a-eea73fdd9d01-iptables-alerter-script" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.516183 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d834c2a-f008-492e-9aac-362fe7a73672" volumeName="kubernetes.io/secret/7d834c2a-f008-492e-9aac-362fe7a73672-console-serving-cert" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.516238 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2ec1edc-b911-4537-ac00-f708ef6c9062" volumeName="kubernetes.io/configmap/a2ec1edc-b911-4537-ac00-f708ef6c9062-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.516267 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d205185b-fc45-4a76-9719-20b00107abf1" volumeName="kubernetes.io/configmap/d205185b-fc45-4a76-9719-20b00107abf1-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.516293 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d332bbb4-c699-44e8-96ba-516d8ca03e95" volumeName="kubernetes.io/configmap/d332bbb4-c699-44e8-96ba-516d8ca03e95-ovnkube-config" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.516328 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4214a36a-fa6a-4290-aadc-2336881f84ba" volumeName="kubernetes.io/secret/4214a36a-fa6a-4290-aadc-2336881f84ba-metrics-tls" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.516340 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="806dfce0-6207-4feb-bee4-ce73cfd4b552" volumeName="kubernetes.io/empty-dir/806dfce0-6207-4feb-bee4-ce73cfd4b552-node-exporter-textfile" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.516361 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b981f97a-f238-4ef6-bbc8-e66cdd90dc90" volumeName="kubernetes.io/projected/b981f97a-f238-4ef6-bbc8-e66cdd90dc90-kube-api-access-hzhgj" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.516381 6221 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d205185b-fc45-4a76-9719-20b00107abf1" volumeName="kubernetes.io/empty-dir/d205185b-fc45-4a76-9719-20b00107abf1-alertmanager-main-db" seLinuxMountContext="" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.516394 6221 reconstruct.go:97] "Volume reconstruction finished" Oct 30 01:17:56.517320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.516401 6221 reconciler.go:29] "Reconciler: start to sync state" Oct 30 01:17:56.537503 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.537143 6221 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 01:17:56.537682 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.537641 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2vsns_d332bbb4-c699-44e8-96ba-516d8ca03e95/ovn-acl-logging/0.log" Oct 30 01:17:56.539145 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.539125 6221 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 01:17:56.539145 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.539148 6221 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 01:17:56.539323 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.539167 6221 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 01:17:56.539323 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:17:56.539206 6221 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 01:17:56.540860 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.540841 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2vsns_d332bbb4-c699-44e8-96ba-516d8ca03e95/ovn-acl-logging/0.log" Oct 30 01:17:56.552287 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.552265 6221 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:17:56.556868 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.556839 6221 generic.go:356] "Generic (PLEG): container finished" podID="d205185b-fc45-4a76-9719-20b00107abf1" containerID="fa086018fcdc7e625873376ddc1edd48c8c59518991ac6ffee80dc216b9ef4af" exitCode=0 Oct 30 01:17:56.563748 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.563727 6221 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 01:17:56.565592 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.565573 6221 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:17:56.566393 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.566374 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2vsns_d332bbb4-c699-44e8-96ba-516d8ca03e95/ovn-acl-logging/0.log" Oct 30 01:17:56.566778 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.566750 6221 generic.go:356] "Generic (PLEG): container finished" podID="d332bbb4-c699-44e8-96ba-516d8ca03e95" containerID="c9e1e3083fc57e5d20d155273f7f436856af4d4f9acdf10b090e03b8c4a2a826" exitCode=1 Oct 30 01:17:56.572477 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.572449 6221 generic.go:356] "Generic (PLEG): container finished" podID="806dfce0-6207-4feb-bee4-ce73cfd4b552" containerID="2f21aa0a1d97280e413497f2f866bc7808fbc92d642ad487b104520c4005567a" exitCode=0 Oct 30 01:17:56.577255 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.577235 6221 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:17:56.577406 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.577388 6221 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 01:17:56.577482 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.577413 6221 generic.go:356] "Generic (PLEG): container finished" podID="a707a897ca34d8ee2c04b93cf900a7fe" containerID="f650c53e0fbc5e060c63f8fe0b1a3717f4011b1c615b62d79cfb747b475ff9db" exitCode=0 Oct 30 01:17:56.593620 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.593578 6221 generic.go:356] "Generic (PLEG): container finished" podID="a2ec1edc-b911-4537-ac00-f708ef6c9062" containerID="370e8a1f6e0cbe68cc5e92bc862b06aef94633f5234e2492c490210f9b65c831" exitCode=0 Oct 30 01:17:56.593620 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.593608 6221 generic.go:356] "Generic (PLEG): container finished" podID="a2ec1edc-b911-4537-ac00-f708ef6c9062" containerID="4b2a2aae230e0151dc4b9d8ff9bcfd3f70c67b08a1050ad1bd1607d51c156785" exitCode=0 Oct 30 01:17:56.593620 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.593618 6221 generic.go:356] "Generic (PLEG): container finished" podID="a2ec1edc-b911-4537-ac00-f708ef6c9062" containerID="727d892a7f2f45f72d36529dfc6d7e79d888779eee6fce1084b3d43c77bdeb7c" exitCode=0 Oct 30 01:17:56.593843 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.593626 6221 generic.go:356] "Generic (PLEG): container finished" podID="a2ec1edc-b911-4537-ac00-f708ef6c9062" containerID="24f1c9127a58360755dca345cc6330058a4b89b515cb75b47698047f6165dfb1" exitCode=0 Oct 30 01:17:56.593843 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.593634 6221 generic.go:356] "Generic (PLEG): container finished" podID="a2ec1edc-b911-4537-ac00-f708ef6c9062" containerID="039acf50b948637d60e6123a560a9feeb4775d2977aeddf9fd23a3a1cf164c16" exitCode=0 Oct 30 01:17:56.593843 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.593642 6221 generic.go:356] "Generic (PLEG): container finished" podID="a2ec1edc-b911-4537-ac00-f708ef6c9062" containerID="0cf075b35ef887db402a9fc1eb9f571aa9debf1d1350ead6acd56e93d573d921" exitCode=0 Oct 30 01:17:56.611477 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.611461 6221 manager.go:324] Recovery completed Oct 30 01:17:56.633613 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.633522 6221 cpu_manager.go:226] "Starting" policy="none" Oct 30 01:17:56.633613 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.633563 6221 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 01:17:56.633613 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.633581 6221 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 01:17:56.633788 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.633696 6221 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 01:17:56.633788 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.633706 6221 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 01:17:56.633788 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.633724 6221 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 01:17:56.633788 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.633729 6221 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 01:17:56.633788 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.633734 6221 policy_none.go:50] "Start" Oct 30 01:17:56.633788 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.633742 6221 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 01:17:56.633788 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.633751 6221 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 01:17:56.633969 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.633873 6221 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 01:17:56.633969 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.633880 6221 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 01:17:56.633969 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.633885 6221 policy_none.go:44] "Start" Oct 30 01:17:56.638318 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.638292 6221 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 01:17:56.640199 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:17:56.640178 6221 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 01:17:56.640825 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.640812 6221 manager.go:341] "Starting Device Plugin manager" Oct 30 01:17:56.641334 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:17:56.640838 6221 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 01:17:56.641371 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.641338 6221 server.go:87] "Starting device plugin registration server" Oct 30 01:17:56.641631 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.641620 6221 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 01:17:56.641674 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.641633 6221 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 01:17:56.641753 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.641735 6221 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 01:17:56.641887 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.641851 6221 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 01:17:56.641946 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.641896 6221 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 01:17:56.641946 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.641903 6221 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 01:17:56.642039 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.641956 6221 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T01:17:56.641885027Z","UUID":"f599c904-cc2e-45bd-8d33-352e81acf1f3","Handler":null,"Name":"","Endpoint":""} Oct 30 01:17:56.643179 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.643161 6221 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 01:17:56.643317 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.643188 6221 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 01:17:56.643645 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:17:56.643463 6221 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 01:17:56.742543 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.742510 6221 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 01:17:56.745771 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.745753 6221 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-65.ec2.internal" event="NodeHasSufficientMemory" Oct 30 01:17:56.745841 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.745785 6221 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-65.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 01:17:56.745841 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.745795 6221 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-135-65.ec2.internal" event="NodeHasSufficientPID" Oct 30 01:17:56.745841 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.745836 6221 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-135-65.ec2.internal" Oct 30 01:17:56.798501 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.798481 6221 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-135-65.ec2.internal" Oct 30 01:17:56.798580 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.798568 6221 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-135-65.ec2.internal" Oct 30 01:17:56.840324 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.840279 6221 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-65.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-65.ec2.internal"] Oct 30 01:17:56.840989 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.840963 6221 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-65.ec2.internal" Oct 30 01:17:56.840989 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.840963 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-65.ec2.internal" event={"ID":"7af5b4d87431a44e4b7050910f6ad38b","Type":"ContainerStarted","Data":"24c74d59abeef83fa778f4eda6f5b24e11272bf67efc350f917a8284348edfac"} Oct 30 01:17:56.841138 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.841009 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-65.ec2.internal" event={"ID":"7af5b4d87431a44e4b7050910f6ad38b","Type":"ContainerStarted","Data":"415e62159ff51389db9ac1f6f638080802e928cef05b934bed5ae6c9b0e64520"} Oct 30 01:17:56.841138 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.841029 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-65.ec2.internal" event={"ID":"a707a897ca34d8ee2c04b93cf900a7fe","Type":"ContainerStarted","Data":"d36d752ffc2bfa45f23cdac39293fef6b23fee8e9a1f7aa39f0afe1d235a1e0f"} Oct 30 01:17:56.841138 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.841047 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-65.ec2.internal" event={"ID":"a707a897ca34d8ee2c04b93cf900a7fe","Type":"ContainerDied","Data":"f650c53e0fbc5e060c63f8fe0b1a3717f4011b1c615b62d79cfb747b475ff9db"} Oct 30 01:17:56.841138 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.841066 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-65.ec2.internal" event={"ID":"a707a897ca34d8ee2c04b93cf900a7fe","Type":"ContainerStarted","Data":"e357503c9c89fb4c2823d4997882a3185c18d2abb7aa618b6b31f657e077742d"} Oct 30 01:17:56.841303 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.841152 6221 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-65.ec2.internal" Oct 30 01:17:56.874599 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.874572 6221 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 01:17:56.874704 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:17:56.874623 6221 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-65.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-65.ec2.internal" Oct 30 01:17:56.875189 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.875170 6221 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 01:17:56.875286 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:17:56.875218 6221 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-65.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-65.ec2.internal" Oct 30 01:17:56.965593 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.965500 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7af5b4d87431a44e4b7050910f6ad38b-config\") pod \"kube-apiserver-proxy-ip-10-0-135-65.ec2.internal\" (UID: \"7af5b4d87431a44e4b7050910f6ad38b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-65.ec2.internal" Oct 30 01:17:56.965593 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.965559 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a707a897ca34d8ee2c04b93cf900a7fe-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-65.ec2.internal\" (UID: \"a707a897ca34d8ee2c04b93cf900a7fe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-65.ec2.internal" Oct 30 01:17:56.965738 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:56.965617 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a707a897ca34d8ee2c04b93cf900a7fe-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-65.ec2.internal\" (UID: \"a707a897ca34d8ee2c04b93cf900a7fe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-65.ec2.internal" Oct 30 01:17:57.066029 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.065997 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7af5b4d87431a44e4b7050910f6ad38b-config\") pod \"kube-apiserver-proxy-ip-10-0-135-65.ec2.internal\" (UID: \"7af5b4d87431a44e4b7050910f6ad38b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-65.ec2.internal" Oct 30 01:17:57.066029 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.066030 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a707a897ca34d8ee2c04b93cf900a7fe-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-65.ec2.internal\" (UID: \"a707a897ca34d8ee2c04b93cf900a7fe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-65.ec2.internal" Oct 30 01:17:57.066209 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.066048 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a707a897ca34d8ee2c04b93cf900a7fe-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-65.ec2.internal\" (UID: \"a707a897ca34d8ee2c04b93cf900a7fe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-65.ec2.internal" Oct 30 01:17:57.066209 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.066098 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a707a897ca34d8ee2c04b93cf900a7fe-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-65.ec2.internal\" (UID: \"a707a897ca34d8ee2c04b93cf900a7fe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-65.ec2.internal" Oct 30 01:17:57.066209 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.066114 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a707a897ca34d8ee2c04b93cf900a7fe-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-65.ec2.internal\" (UID: \"a707a897ca34d8ee2c04b93cf900a7fe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-65.ec2.internal" Oct 30 01:17:57.066209 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.066116 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7af5b4d87431a44e4b7050910f6ad38b-config\") pod \"kube-apiserver-proxy-ip-10-0-135-65.ec2.internal\" (UID: \"7af5b4d87431a44e4b7050910f6ad38b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-65.ec2.internal" Oct 30 01:17:57.491635 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.491590 6221 apiserver.go:52] "Watching apiserver" Oct 30 01:17:57.543920 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.543886 6221 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 01:17:57.544581 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.544556 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-9bf55565f-w7bkn","openshift-monitoring/prometheus-operator-9cd89b74c-8qcgr","kube-system/kube-apiserver-proxy-ip-10-0-135-65.ec2.internal","openshift-network-operator/iptables-alerter-mdwqw","openshift-console/console-7b6fbb6f6f-9z4ws","openshift-console/console-9477b4757-pfqgz","openshift-multus/multus-additional-cni-plugins-xfvmk","kube-system/konnectivity-agent-9f9b4","openshift-dns/node-resolver-pf9jv","openshift-monitoring/metrics-server-944876b99-29n9f","openshift-insights/insights-runtime-extractor-hft6p","openshift-multus/network-metrics-daemon-xj8mh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-65.ec2.internal","openshift-monitoring/node-exporter-tdj6l","openshift-network-diagnostics/network-check-target-bbhx5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs","openshift-multus/multus-pgk7v","openshift-image-registry/node-ca-lm86p","openshift-monitoring/alertmanager-main-0","openshift-ovn-kubernetes/ovnkube-node-2vsns","kube-system/global-pull-secret-syncer-dxz45","openshift-dns/dns-default-cwzvk","openshift-ingress-canary/ingress-canary-tk7cl","openshift-cluster-node-tuning-operator/tuned-pkkfl"] Oct 30 01:17:57.561172 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.561141 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-58lhf\"" Oct 30 01:17:57.561306 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.561207 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-n79rv\"" Oct 30 01:17:57.561414 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.561395 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 01:17:57.561487 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.561400 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 01:17:57.561487 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.561444 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 01:17:57.561487 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.561450 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 01:17:57.561487 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.561483 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 01:17:57.561696 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.561500 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 01:17:57.561696 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.561506 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 01:17:57.561696 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.561554 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 01:17:57.561696 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.561578 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 01:17:57.561696 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.561588 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 01:17:57.561696 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.561598 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-srg4h\"" Oct 30 01:17:57.561696 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.561621 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 01:17:57.561696 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.561634 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hb7tb\"" Oct 30 01:17:57.561696 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.561668 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 01:17:57.561998 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.561729 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 01:17:57.561998 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.561738 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j8bsj\"" Oct 30 01:17:57.561998 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.561787 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xp5gx\"" Oct 30 01:17:57.561998 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.561844 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pcscp\"" Oct 30 01:17:57.561998 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.561912 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 01:17:57.561998 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.561914 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 01:17:57.561998 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.561962 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 01:17:57.564847 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.564820 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-p4hjp\"" Oct 30 01:17:57.564847 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.564836 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 01:17:57.574374 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.574350 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 01:17:57.574481 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.574417 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 01:17:57.574481 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.574440 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 01:17:57.574593 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.574493 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 01:17:57.574710 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.574695 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-fsgft\"" Oct 30 01:17:57.574815 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.574799 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-wjscp\"" Oct 30 01:17:57.574815 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.574806 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 01:17:57.574815 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.574808 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 01:17:57.574815 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.574825 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 01:17:57.575088 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.575005 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 01:17:57.575088 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.575010 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 01:17:57.575088 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.575059 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 01:17:57.575088 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.575066 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 01:17:57.575088 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.575081 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 01:17:57.575347 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.575105 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-gpws7\"" Oct 30 01:17:57.575347 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.575115 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 01:17:57.575347 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.575106 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 01:17:57.575347 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.575150 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 01:17:57.575347 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.575155 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 01:17:57.575347 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.575167 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 01:17:57.575347 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.575177 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 01:17:57.575347 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.575206 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 01:17:57.575347 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.575230 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 01:17:57.575347 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.575258 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 01:17:57.575347 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.575268 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 01:17:57.575347 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.575290 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 01:17:57.575347 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.575292 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 01:17:57.575778 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.575355 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 01:17:57.575778 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.575358 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 01:17:57.575778 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.575373 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 01:17:57.575778 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.575401 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-p2b59\"" Oct 30 01:17:57.575778 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.575438 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vthnr\"" Oct 30 01:17:57.575778 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.575441 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k4mfm\"" Oct 30 01:17:57.575778 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.575443 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7dsj7\"" Oct 30 01:17:57.575778 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.575458 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-aeu5o81g4aq44\"" Oct 30 01:17:57.575778 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.575463 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 01:17:57.575778 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.575467 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-bvfsl\"" Oct 30 01:17:57.575778 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.575496 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 01:17:57.598875 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.598854 6221 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-65.ec2.internal" Oct 30 01:17:57.598980 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.598932 6221 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-65.ec2.internal" Oct 30 01:17:57.612740 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.612717 6221 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 01:17:57.622674 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.622656 6221 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 01:17:57.622760 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:17:57.622702 6221 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-135-65.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-65.ec2.internal" Oct 30 01:17:57.623398 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.623382 6221 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 01:17:57.623459 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:17:57.623424 6221 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-135-65.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-65.ec2.internal" Oct 30 01:17:57.660778 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.660756 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 01:17:57.660890 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.660803 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 01:17:57.660890 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.660811 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 01:17:57.660890 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.660832 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 01:17:57.661134 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.661117 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ms58v\"" Oct 30 01:17:57.661276 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.661257 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 01:17:57.661348 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.661286 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 01:17:57.661348 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.661334 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 01:17:57.661348 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.661342 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 01:17:57.661565 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.661527 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-54zkd\"" Oct 30 01:17:57.661654 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.661608 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 01:17:57.661654 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.661647 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 01:17:57.661733 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.661662 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 01:17:57.661733 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.661681 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-nbfct\"" Oct 30 01:17:57.661802 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.661783 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 01:17:57.661858 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.661806 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 01:17:57.661858 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.661816 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 01:17:57.661944 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.661922 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 01:17:57.662218 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.662201 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 01:17:57.662282 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.662224 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 01:17:57.662282 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.662224 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xmk59\"" Oct 30 01:17:57.662282 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.662245 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 01:17:57.662282 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.662250 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 01:17:57.662282 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.662206 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 01:17:57.662518 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.662505 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 01:17:57.668539 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.668515 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-sys\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:17:57.668594 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.668556 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/06f5cece-5273-44d1-90ba-67bf5dc0395b-crio-socket\") pod \"insights-runtime-extractor-hft6p\" (UID: \"06f5cece-5273-44d1-90ba-67bf5dc0395b\") " pod="openshift-insights/insights-runtime-extractor-hft6p" Oct 30 01:17:57.668594 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.668574 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2ec1edc-b911-4537-ac00-f708ef6c9062-system-cni-dir\") pod \"multus-additional-cni-plugins-xfvmk\" (UID: \"a2ec1edc-b911-4537-ac00-f708ef6c9062\") " pod="openshift-multus/multus-additional-cni-plugins-xfvmk" Oct 30 01:17:57.668657 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.668593 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2ec1edc-b911-4537-ac00-f708ef6c9062-cni-binary-copy\") pod \"multus-additional-cni-plugins-xfvmk\" (UID: \"a2ec1edc-b911-4537-ac00-f708ef6c9062\") " pod="openshift-multus/multus-additional-cni-plugins-xfvmk" Oct 30 01:17:57.668657 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.668646 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-hostroot\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:17:57.668763 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.668744 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bda81d83-62da-4f40-b668-29e2b0e51f35-multus-daemon-config\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:17:57.668833 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.668816 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a4365b4f-a9d0-41eb-9d89-26591d3e3ed5-sys-fs\") pod \"aws-ebs-csi-driver-node-2wbfs\" (UID: \"a4365b4f-a9d0-41eb-9d89-26591d3e3ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs" Oct 30 01:17:57.668863 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.668837 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2ec1edc-b911-4537-ac00-f708ef6c9062-cni-binary-copy\") pod \"multus-additional-cni-plugins-xfvmk\" (UID: \"a2ec1edc-b911-4537-ac00-f708ef6c9062\") " pod="openshift-multus/multus-additional-cni-plugins-xfvmk" Oct 30 01:17:57.668891 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.668866 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d205185b-fc45-4a76-9719-20b00107abf1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:57.668931 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.668920 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d834c2a-f008-492e-9aac-362fe7a73672-trusted-ca-bundle\") pod \"console-9477b4757-pfqgz\" (UID: \"7d834c2a-f008-492e-9aac-362fe7a73672\") " pod="openshift-console/console-9477b4757-pfqgz" Oct 30 01:17:57.668975 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.668959 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bda81d83-62da-4f40-b668-29e2b0e51f35-multus-daemon-config\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:17:57.669005 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.668985 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qbvg\" (UniqueName: \"kubernetes.io/projected/a2ec1edc-b911-4537-ac00-f708ef6c9062-kube-api-access-8qbvg\") pod \"multus-additional-cni-plugins-xfvmk\" (UID: \"a2ec1edc-b911-4537-ac00-f708ef6c9062\") " pod="openshift-multus/multus-additional-cni-plugins-xfvmk" Oct 30 01:17:57.669033 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.669009 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f10aab9c-d4e6-45b5-a703-146114a31329-dbus\") pod \"global-pull-secret-syncer-dxz45\" (UID: \"f10aab9c-d4e6-45b5-a703-146114a31329\") " pod="kube-system/global-pull-secret-syncer-dxz45" Oct 30 01:17:57.669033 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.669027 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f67fc\" (UniqueName: \"kubernetes.io/projected/a4365b4f-a9d0-41eb-9d89-26591d3e3ed5-kube-api-access-f67fc\") pod \"aws-ebs-csi-driver-node-2wbfs\" (UID: \"a4365b4f-a9d0-41eb-9d89-26591d3e3ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs" Oct 30 01:17:57.669093 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.669077 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-lib-modules\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:17:57.669133 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.669118 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e09d6f2e-7cd5-47c3-b18b-482726067e90-etc-tuned\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:17:57.669189 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.669174 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a4365b4f-a9d0-41eb-9d89-26591d3e3ed5-registration-dir\") pod \"aws-ebs-csi-driver-node-2wbfs\" (UID: \"a4365b4f-a9d0-41eb-9d89-26591d3e3ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs" Oct 30 01:17:57.669220 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.669187 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e09d6f2e-7cd5-47c3-b18b-482726067e90-etc-tuned\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:17:57.669220 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.669206 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db13cab2-9b76-44b5-8577-9f8f0d3f292e-console-serving-cert\") pod \"console-7b6fbb6f6f-9z4ws\" (UID: \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\") " pod="openshift-console/console-7b6fbb6f6f-9z4ws" Oct 30 01:17:57.669279 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.669233 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d834c2a-f008-492e-9aac-362fe7a73672-console-serving-cert\") pod \"console-9477b4757-pfqgz\" (UID: \"7d834c2a-f008-492e-9aac-362fe7a73672\") " pod="openshift-console/console-9477b4757-pfqgz" Oct 30 01:17:57.669279 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.669271 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-run-systemd\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.669336 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.669293 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-host-run-ovn-kubernetes\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.669336 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.669320 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d332bbb4-c699-44e8-96ba-516d8ca03e95-ovn-node-metrics-cert\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.669475 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.669451 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-cnibin\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:17:57.669475 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.669454 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d834c2a-f008-492e-9aac-362fe7a73672-console-serving-cert\") pod \"console-9477b4757-pfqgz\" (UID: \"7d834c2a-f008-492e-9aac-362fe7a73672\") " pod="openshift-console/console-9477b4757-pfqgz" Oct 30 01:17:57.669553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.669496 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db13cab2-9b76-44b5-8577-9f8f0d3f292e-console-serving-cert\") pod \"console-7b6fbb6f6f-9z4ws\" (UID: \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\") " pod="openshift-console/console-7b6fbb6f6f-9z4ws" Oct 30 01:17:57.669553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.669504 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8e7e744e-9806-48ca-9d02-94bb07752f36-oauth-serving-cert\") pod \"console-9bf55565f-w7bkn\" (UID: \"8e7e744e-9806-48ca-9d02-94bb07752f36\") " pod="openshift-console/console-9bf55565f-w7bkn" Oct 30 01:17:57.669615 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.669573 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjqdw\" (UniqueName: \"kubernetes.io/projected/380432b5-86bc-4dd3-a846-d7875d621a36-kube-api-access-wjqdw\") pod \"metrics-server-944876b99-29n9f\" (UID: \"380432b5-86bc-4dd3-a846-d7875d621a36\") " pod="openshift-monitoring/metrics-server-944876b99-29n9f" Oct 30 01:17:57.669663 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.669612 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5618fce7-e2ca-4423-887a-eea73fdd9d01-host-slash\") pod \"iptables-alerter-mdwqw\" (UID: \"5618fce7-e2ca-4423-887a-eea73fdd9d01\") " pod="openshift-network-operator/iptables-alerter-mdwqw" Oct 30 01:17:57.669663 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.669620 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d332bbb4-c699-44e8-96ba-516d8ca03e95-ovn-node-metrics-cert\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.669663 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.669640 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-host-cni-netd\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.669756 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.669678 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd48f\" (UniqueName: \"kubernetes.io/projected/06f5cece-5273-44d1-90ba-67bf5dc0395b-kube-api-access-jd48f\") pod \"insights-runtime-extractor-hft6p\" (UID: \"06f5cece-5273-44d1-90ba-67bf5dc0395b\") " pod="openshift-insights/insights-runtime-extractor-hft6p" Oct 30 01:17:57.669756 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.669700 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-host-var-lib-kubelet\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:17:57.669756 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.669717 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzhgj\" (UniqueName: \"kubernetes.io/projected/b981f97a-f238-4ef6-bbc8-e66cdd90dc90-kube-api-access-hzhgj\") pod \"ingress-canary-tk7cl\" (UID: \"b981f97a-f238-4ef6-bbc8-e66cdd90dc90\") " pod="openshift-ingress-canary/ingress-canary-tk7cl" Oct 30 01:17:57.669756 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.669722 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8e7e744e-9806-48ca-9d02-94bb07752f36-oauth-serving-cert\") pod \"console-9bf55565f-w7bkn\" (UID: \"8e7e744e-9806-48ca-9d02-94bb07752f36\") " pod="openshift-console/console-9bf55565f-w7bkn" Oct 30 01:17:57.669756 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.669732 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8e7e744e-9806-48ca-9d02-94bb07752f36-service-ca\") pod \"console-9bf55565f-w7bkn\" (UID: \"8e7e744e-9806-48ca-9d02-94bb07752f36\") " pod="openshift-console/console-9bf55565f-w7bkn" Oct 30 01:17:57.669756 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.669754 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/806dfce0-6207-4feb-bee4-ce73cfd4b552-node-exporter-tls\") pod \"node-exporter-tdj6l\" (UID: \"806dfce0-6207-4feb-bee4-ce73cfd4b552\") " pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:57.670077 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.669769 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a4365b4f-a9d0-41eb-9d89-26591d3e3ed5-device-dir\") pod \"aws-ebs-csi-driver-node-2wbfs\" (UID: \"a4365b4f-a9d0-41eb-9d89-26591d3e3ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs" Oct 30 01:17:57.670077 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.669786 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgt8r\" (UniqueName: \"kubernetes.io/projected/806dfce0-6207-4feb-bee4-ce73cfd4b552-kube-api-access-bgt8r\") pod \"node-exporter-tdj6l\" (UID: \"806dfce0-6207-4feb-bee4-ce73cfd4b552\") " pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:57.670077 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.669805 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/380432b5-86bc-4dd3-a846-d7875d621a36-secret-metrics-server-tls\") pod \"metrics-server-944876b99-29n9f\" (UID: \"380432b5-86bc-4dd3-a846-d7875d621a36\") " pod="openshift-monitoring/metrics-server-944876b99-29n9f" Oct 30 01:17:57.670077 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.669841 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d205185b-fc45-4a76-9719-20b00107abf1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:57.670077 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.669868 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-run-openvswitch\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.670077 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.669904 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8e7e744e-9806-48ca-9d02-94bb07752f36-service-ca\") pod \"console-9bf55565f-w7bkn\" (UID: \"8e7e744e-9806-48ca-9d02-94bb07752f36\") " pod="openshift-console/console-9bf55565f-w7bkn" Oct 30 01:17:57.670077 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.669921 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-host-cni-bin\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.670077 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.669969 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d332bbb4-c699-44e8-96ba-516d8ca03e95-ovnkube-script-lib\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.670077 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.669991 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94864d90-1dd8-472e-a414-29c6b6a4f8f6-metrics-certs\") pod \"network-metrics-daemon-xj8mh\" (UID: \"94864d90-1dd8-472e-a414-29c6b6a4f8f6\") " pod="openshift-multus/network-metrics-daemon-xj8mh" Oct 30 01:17:57.670077 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.670028 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/380432b5-86bc-4dd3-a846-d7875d621a36-secret-metrics-server-tls\") pod \"metrics-server-944876b99-29n9f\" (UID: \"380432b5-86bc-4dd3-a846-d7875d621a36\") " pod="openshift-monitoring/metrics-server-944876b99-29n9f" Oct 30 01:17:57.670077 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.670067 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d205185b-fc45-4a76-9719-20b00107abf1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:57.670406 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.670177 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db13cab2-9b76-44b5-8577-9f8f0d3f292e-oauth-serving-cert\") pod \"console-7b6fbb6f6f-9z4ws\" (UID: \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\") " pod="openshift-console/console-7b6fbb6f6f-9z4ws" Oct 30 01:17:57.670406 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.670200 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94864d90-1dd8-472e-a414-29c6b6a4f8f6-metrics-certs\") pod \"network-metrics-daemon-xj8mh\" (UID: \"94864d90-1dd8-472e-a414-29c6b6a4f8f6\") " pod="openshift-multus/network-metrics-daemon-xj8mh" Oct 30 01:17:57.670406 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.670213 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d205185b-fc45-4a76-9719-20b00107abf1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:57.670406 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.670200 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/806dfce0-6207-4feb-bee4-ce73cfd4b552-node-exporter-tls\") pod \"node-exporter-tdj6l\" (UID: \"806dfce0-6207-4feb-bee4-ce73cfd4b552\") " pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:57.670406 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.670265 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-host-slash\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.670406 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.670272 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d332bbb4-c699-44e8-96ba-516d8ca03e95-ovnkube-script-lib\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.670406 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.670294 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db13cab2-9b76-44b5-8577-9f8f0d3f292e-console-config\") pod \"console-7b6fbb6f6f-9z4ws\" (UID: \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\") " pod="openshift-console/console-7b6fbb6f6f-9z4ws" Oct 30 01:17:57.670406 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.670326 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4214a36a-fa6a-4290-aadc-2336881f84ba-metrics-tls\") pod \"dns-default-cwzvk\" (UID: \"4214a36a-fa6a-4290-aadc-2336881f84ba\") " pod="openshift-dns/dns-default-cwzvk" Oct 30 01:17:57.670406 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.670371 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db13cab2-9b76-44b5-8577-9f8f0d3f292e-oauth-serving-cert\") pod \"console-7b6fbb6f6f-9z4ws\" (UID: \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\") " pod="openshift-console/console-7b6fbb6f6f-9z4ws" Oct 30 01:17:57.670406 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.670393 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d205185b-fc45-4a76-9719-20b00107abf1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:57.670406 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.670399 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a2ec1edc-b911-4537-ac00-f708ef6c9062-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xfvmk\" (UID: \"a2ec1edc-b911-4537-ac00-f708ef6c9062\") " pod="openshift-multus/multus-additional-cni-plugins-xfvmk" Oct 30 01:17:57.670877 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.670434 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e30ba693-6284-485e-95fd-ad639627aa8d-konnectivity-ca\") pod \"konnectivity-agent-9f9b4\" (UID: \"e30ba693-6284-485e-95fd-ad639627aa8d\") " pod="kube-system/konnectivity-agent-9f9b4" Oct 30 01:17:57.670877 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.670465 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a4365b4f-a9d0-41eb-9d89-26591d3e3ed5-socket-dir\") pod \"aws-ebs-csi-driver-node-2wbfs\" (UID: \"a4365b4f-a9d0-41eb-9d89-26591d3e3ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs" Oct 30 01:17:57.670877 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.670490 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4214a36a-fa6a-4290-aadc-2336881f84ba-metrics-tls\") pod \"dns-default-cwzvk\" (UID: \"4214a36a-fa6a-4290-aadc-2336881f84ba\") " pod="openshift-dns/dns-default-cwzvk" Oct 30 01:17:57.670877 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.670490 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a4365b4f-a9d0-41eb-9d89-26591d3e3ed5-etc-selinux\") pod \"aws-ebs-csi-driver-node-2wbfs\" (UID: \"a4365b4f-a9d0-41eb-9d89-26591d3e3ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs" Oct 30 01:17:57.670877 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.670561 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a2ec1edc-b911-4537-ac00-f708ef6c9062-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xfvmk\" (UID: \"a2ec1edc-b911-4537-ac00-f708ef6c9062\") " pod="openshift-multus/multus-additional-cni-plugins-xfvmk" Oct 30 01:17:57.670877 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.670575 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e09d6f2e-7cd5-47c3-b18b-482726067e90-tmp\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:17:57.670877 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.670614 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d205185b-fc45-4a76-9719-20b00107abf1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:57.670877 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.670632 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e30ba693-6284-485e-95fd-ad639627aa8d-konnectivity-ca\") pod \"konnectivity-agent-9f9b4\" (UID: \"e30ba693-6284-485e-95fd-ad639627aa8d\") " pod="kube-system/konnectivity-agent-9f9b4" Oct 30 01:17:57.670877 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.670666 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e09d6f2e-7cd5-47c3-b18b-482726067e90-tmp\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:17:57.670877 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.670711 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/806dfce0-6207-4feb-bee4-ce73cfd4b552-sys\") pod \"node-exporter-tdj6l\" (UID: \"806dfce0-6207-4feb-bee4-ce73cfd4b552\") " pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:57.670877 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.670742 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d205185b-fc45-4a76-9719-20b00107abf1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:57.670877 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.670750 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/806dfce0-6207-4feb-bee4-ce73cfd4b552-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tdj6l\" (UID: \"806dfce0-6207-4feb-bee4-ce73cfd4b552\") " pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:57.670877 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.670782 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmptt\" (UniqueName: \"kubernetes.io/projected/e09d6f2e-7cd5-47c3-b18b-482726067e90-kube-api-access-nmptt\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:17:57.670877 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.670818 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-multus-socket-dir-parent\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:17:57.670877 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.670877 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db13cab2-9b76-44b5-8577-9f8f0d3f292e-console-config\") pod \"console-7b6fbb6f6f-9z4ws\" (UID: \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\") " pod="openshift-console/console-7b6fbb6f6f-9z4ws" Oct 30 01:17:57.671525 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.670924 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdvvv\" (UniqueName: \"kubernetes.io/projected/ecb9d3db-582f-41a0-945d-7bd7e4892ff8-kube-api-access-kdvvv\") pod \"node-ca-lm86p\" (UID: \"ecb9d3db-582f-41a0-945d-7bd7e4892ff8\") " pod="openshift-image-registry/node-ca-lm86p" Oct 30 01:17:57.671525 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.670938 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/806dfce0-6207-4feb-bee4-ce73cfd4b552-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-tdj6l\" (UID: \"806dfce0-6207-4feb-bee4-ce73cfd4b552\") " pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:57.671525 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.670966 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-etc-systemd\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:17:57.671525 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.671070 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f10aab9c-d4e6-45b5-a703-146114a31329-original-pull-secret\") pod \"global-pull-secret-syncer-dxz45\" (UID: \"f10aab9c-d4e6-45b5-a703-146114a31329\") " pod="kube-system/global-pull-secret-syncer-dxz45" Oct 30 01:17:57.671525 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.671109 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e7e744e-9806-48ca-9d02-94bb07752f36-console-serving-cert\") pod \"console-9bf55565f-w7bkn\" (UID: \"8e7e744e-9806-48ca-9d02-94bb07752f36\") " pod="openshift-console/console-9bf55565f-w7bkn" Oct 30 01:17:57.671525 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.671136 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db13cab2-9b76-44b5-8577-9f8f0d3f292e-tmp\") pod \"console-7b6fbb6f6f-9z4ws\" (UID: \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\") " pod="openshift-console/console-7b6fbb6f6f-9z4ws" Oct 30 01:17:57.671525 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.671153 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-etc-sysctl-d\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:17:57.671525 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.671168 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-var-lib-kubelet\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:17:57.671525 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.671190 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89k2s\" (UniqueName: \"kubernetes.io/projected/f4602ac2-652b-4026-9a0d-d72162bb3258-kube-api-access-89k2s\") pod \"network-check-target-bbhx5\" (UID: \"f4602ac2-652b-4026-9a0d-d72162bb3258\") " pod="openshift-network-diagnostics/network-check-target-bbhx5" Oct 30 01:17:57.671525 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.671250 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db13cab2-9b76-44b5-8577-9f8f0d3f292e-tmp\") pod \"console-7b6fbb6f6f-9z4ws\" (UID: \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\") " pod="openshift-console/console-7b6fbb6f6f-9z4ws" Oct 30 01:17:57.671525 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.671285 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd9jh\" (UniqueName: \"kubernetes.io/projected/7d834c2a-f008-492e-9aac-362fe7a73672-kube-api-access-sd9jh\") pod \"console-9477b4757-pfqgz\" (UID: \"7d834c2a-f008-492e-9aac-362fe7a73672\") " pod="openshift-console/console-9477b4757-pfqgz" Oct 30 01:17:57.671525 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.671314 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5618fce7-e2ca-4423-887a-eea73fdd9d01-iptables-alerter-script\") pod \"iptables-alerter-mdwqw\" (UID: \"5618fce7-e2ca-4423-887a-eea73fdd9d01\") " pod="openshift-network-operator/iptables-alerter-mdwqw" Oct 30 01:17:57.671525 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.671341 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-run\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:17:57.671525 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.671368 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/06f5cece-5273-44d1-90ba-67bf5dc0395b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hft6p\" (UID: \"06f5cece-5273-44d1-90ba-67bf5dc0395b\") " pod="openshift-insights/insights-runtime-extractor-hft6p" Oct 30 01:17:57.671525 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.671380 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e7e744e-9806-48ca-9d02-94bb07752f36-console-serving-cert\") pod \"console-9bf55565f-w7bkn\" (UID: \"8e7e744e-9806-48ca-9d02-94bb07752f36\") " pod="openshift-console/console-9bf55565f-w7bkn" Oct 30 01:17:57.671525 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.671421 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f10aab9c-d4e6-45b5-a703-146114a31329-original-pull-secret\") pod \"global-pull-secret-syncer-dxz45\" (UID: \"f10aab9c-d4e6-45b5-a703-146114a31329\") " pod="kube-system/global-pull-secret-syncer-dxz45" Oct 30 01:17:57.671525 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.671525 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcd88\" (UniqueName: \"kubernetes.io/projected/d205185b-fc45-4a76-9719-20b00107abf1-kube-api-access-mcd88\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:57.672126 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.671567 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d834c2a-f008-492e-9aac-362fe7a73672-tmp\") pod \"console-9477b4757-pfqgz\" (UID: \"7d834c2a-f008-492e-9aac-362fe7a73672\") " pod="openshift-console/console-9477b4757-pfqgz" Oct 30 01:17:57.672126 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.671559 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5618fce7-e2ca-4423-887a-eea73fdd9d01-iptables-alerter-script\") pod \"iptables-alerter-mdwqw\" (UID: \"5618fce7-e2ca-4423-887a-eea73fdd9d01\") " pod="openshift-network-operator/iptables-alerter-mdwqw" Oct 30 01:17:57.672126 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.671605 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/06f5cece-5273-44d1-90ba-67bf5dc0395b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hft6p\" (UID: \"06f5cece-5273-44d1-90ba-67bf5dc0395b\") " pod="openshift-insights/insights-runtime-extractor-hft6p" Oct 30 01:17:57.672126 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.671624 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d834c2a-f008-492e-9aac-362fe7a73672-tmp\") pod \"console-9477b4757-pfqgz\" (UID: \"7d834c2a-f008-492e-9aac-362fe7a73672\") " pod="openshift-console/console-9477b4757-pfqgz" Oct 30 01:17:57.672126 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.671700 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d834c2a-f008-492e-9aac-362fe7a73672-oauth-serving-cert\") pod \"console-9477b4757-pfqgz\" (UID: \"7d834c2a-f008-492e-9aac-362fe7a73672\") " pod="openshift-console/console-9477b4757-pfqgz" Oct 30 01:17:57.672126 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.671734 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-etc-openvswitch\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.672126 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.671760 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-run-ovn\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.672126 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.671791 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4214a36a-fa6a-4290-aadc-2336881f84ba-config-volume\") pod \"dns-default-cwzvk\" (UID: \"4214a36a-fa6a-4290-aadc-2336881f84ba\") " pod="openshift-dns/dns-default-cwzvk" Oct 30 01:17:57.672126 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.671826 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-host-run-k8s-cni-cncf-io\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:17:57.672126 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.671831 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d834c2a-f008-492e-9aac-362fe7a73672-oauth-serving-cert\") pod \"console-9477b4757-pfqgz\" (UID: \"7d834c2a-f008-492e-9aac-362fe7a73672\") " pod="openshift-console/console-9477b4757-pfqgz" Oct 30 01:17:57.672126 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.671876 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-host-kubelet\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.672126 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.671920 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-log-socket\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.672126 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.671952 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:57.672126 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.671962 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4214a36a-fa6a-4290-aadc-2336881f84ba-config-volume\") pod \"dns-default-cwzvk\" (UID: \"4214a36a-fa6a-4290-aadc-2336881f84ba\") " pod="openshift-dns/dns-default-cwzvk" Oct 30 01:17:57.672126 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.671975 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bda81d83-62da-4f40-b668-29e2b0e51f35-cni-binary-copy\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:17:57.672126 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.672024 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9b098bdf-b302-4954-907b-eadab89054a0-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-8qcgr\" (UID: \"9b098bdf-b302-4954-907b-eadab89054a0\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8qcgr" Oct 30 01:17:57.672126 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.672071 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/806dfce0-6207-4feb-bee4-ce73cfd4b552-node-exporter-wtmp\") pod \"node-exporter-tdj6l\" (UID: \"806dfce0-6207-4feb-bee4-ce73cfd4b552\") " pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:57.672126 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.672098 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bda81d83-62da-4f40-b668-29e2b0e51f35-cni-binary-copy\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:17:57.672126 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.672124 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db13cab2-9b76-44b5-8577-9f8f0d3f292e-console-oauth-config\") pod \"console-7b6fbb6f6f-9z4ws\" (UID: \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\") " pod="openshift-console/console-7b6fbb6f6f-9z4ws" Oct 30 01:17:57.672874 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.672162 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:57.672874 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.672168 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db13cab2-9b76-44b5-8577-9f8f0d3f292e-trusted-ca-bundle\") pod \"console-7b6fbb6f6f-9z4ws\" (UID: \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\") " pod="openshift-console/console-7b6fbb6f6f-9z4ws" Oct 30 01:17:57.672874 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.672176 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9b098bdf-b302-4954-907b-eadab89054a0-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-8qcgr\" (UID: \"9b098bdf-b302-4954-907b-eadab89054a0\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8qcgr" Oct 30 01:17:57.672874 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.672203 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/06f5cece-5273-44d1-90ba-67bf5dc0395b-data-volume\") pod \"insights-runtime-extractor-hft6p\" (UID: \"06f5cece-5273-44d1-90ba-67bf5dc0395b\") " pod="openshift-insights/insights-runtime-extractor-hft6p" Oct 30 01:17:57.672874 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.672302 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/06f5cece-5273-44d1-90ba-67bf5dc0395b-data-volume\") pod \"insights-runtime-extractor-hft6p\" (UID: \"06f5cece-5273-44d1-90ba-67bf5dc0395b\") " pod="openshift-insights/insights-runtime-extractor-hft6p" Oct 30 01:17:57.672874 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.672331 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:57.672874 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.672348 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-host-run-netns\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:17:57.672874 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.672368 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db13cab2-9b76-44b5-8577-9f8f0d3f292e-console-oauth-config\") pod \"console-7b6fbb6f6f-9z4ws\" (UID: \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\") " pod="openshift-console/console-7b6fbb6f6f-9z4ws" Oct 30 01:17:57.672874 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.672408 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7frp\" (UniqueName: \"kubernetes.io/projected/db13cab2-9b76-44b5-8577-9f8f0d3f292e-kube-api-access-l7frp\") pod \"console-7b6fbb6f6f-9z4ws\" (UID: \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\") " pod="openshift-console/console-7b6fbb6f6f-9z4ws" Oct 30 01:17:57.672874 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.672438 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-etc-kubernetes\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:17:57.672874 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.672469 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-web-config\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:57.672874 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.672481 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:57.672874 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.672497 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d834c2a-f008-492e-9aac-362fe7a73672-console-oauth-config\") pod \"console-9477b4757-pfqgz\" (UID: \"7d834c2a-f008-492e-9aac-362fe7a73672\") " pod="openshift-console/console-9477b4757-pfqgz" Oct 30 01:17:57.672874 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.672514 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-host-run-netns\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.672874 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.672544 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2ec1edc-b911-4537-ac00-f708ef6c9062-cnibin\") pod \"multus-additional-cni-plugins-xfvmk\" (UID: \"a2ec1edc-b911-4537-ac00-f708ef6c9062\") " pod="openshift-multus/multus-additional-cni-plugins-xfvmk" Oct 30 01:17:57.672874 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.672596 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-multus-conf-dir\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:17:57.672874 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.672660 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/806dfce0-6207-4feb-bee4-ce73cfd4b552-node-exporter-accelerators-collector-config\") pod \"node-exporter-tdj6l\" (UID: \"806dfce0-6207-4feb-bee4-ce73cfd4b552\") " pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:57.672874 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.672707 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-web-config\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:57.672874 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.672718 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-etc-sysconfig\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:17:57.672874 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.672747 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d834c2a-f008-492e-9aac-362fe7a73672-console-oauth-config\") pod \"console-9477b4757-pfqgz\" (UID: \"7d834c2a-f008-492e-9aac-362fe7a73672\") " pod="openshift-console/console-9477b4757-pfqgz" Oct 30 01:17:57.672874 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.672842 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dhns\" (UniqueName: \"kubernetes.io/projected/c819c74a-0f8d-4a5e-9e1b-830196f6b117-kube-api-access-4dhns\") pod \"node-resolver-pf9jv\" (UID: \"c819c74a-0f8d-4a5e-9e1b-830196f6b117\") " pod="openshift-dns/node-resolver-pf9jv" Oct 30 01:17:57.672874 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.672849 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/806dfce0-6207-4feb-bee4-ce73cfd4b552-node-exporter-accelerators-collector-config\") pod \"node-exporter-tdj6l\" (UID: \"806dfce0-6207-4feb-bee4-ce73cfd4b552\") " pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:57.673826 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.672886 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a2ec1edc-b911-4537-ac00-f708ef6c9062-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xfvmk\" (UID: \"a2ec1edc-b911-4537-ac00-f708ef6c9062\") " pod="openshift-multus/multus-additional-cni-plugins-xfvmk" Oct 30 01:17:57.673826 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.672915 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8e7e744e-9806-48ca-9d02-94bb07752f36-console-config\") pod \"console-9bf55565f-w7bkn\" (UID: \"8e7e744e-9806-48ca-9d02-94bb07752f36\") " pod="openshift-console/console-9bf55565f-w7bkn" Oct 30 01:17:57.673826 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.672932 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/806dfce0-6207-4feb-bee4-ce73cfd4b552-root\") pod \"node-exporter-tdj6l\" (UID: \"806dfce0-6207-4feb-bee4-ce73cfd4b552\") " pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:57.673826 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.672968 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/380432b5-86bc-4dd3-a846-d7875d621a36-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-944876b99-29n9f\" (UID: \"380432b5-86bc-4dd3-a846-d7875d621a36\") " pod="openshift-monitoring/metrics-server-944876b99-29n9f" Oct 30 01:17:57.673826 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.673050 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b981f97a-f238-4ef6-bbc8-e66cdd90dc90-cert\") pod \"ingress-canary-tk7cl\" (UID: \"b981f97a-f238-4ef6-bbc8-e66cdd90dc90\") " pod="openshift-ingress-canary/ingress-canary-tk7cl" Oct 30 01:17:57.673826 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.673078 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:57.673826 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.673105 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a2ec1edc-b911-4537-ac00-f708ef6c9062-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xfvmk\" (UID: \"a2ec1edc-b911-4537-ac00-f708ef6c9062\") " pod="openshift-multus/multus-additional-cni-plugins-xfvmk" Oct 30 01:17:57.673826 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.673126 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d332bbb4-c699-44e8-96ba-516d8ca03e95-env-overrides\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.673826 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.673155 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rmpm\" (UniqueName: \"kubernetes.io/projected/94864d90-1dd8-472e-a414-29c6b6a4f8f6-kube-api-access-5rmpm\") pod \"network-metrics-daemon-xj8mh\" (UID: \"94864d90-1dd8-472e-a414-29c6b6a4f8f6\") " pod="openshift-multus/network-metrics-daemon-xj8mh" Oct 30 01:17:57.673826 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.673165 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/380432b5-86bc-4dd3-a846-d7875d621a36-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-944876b99-29n9f\" (UID: \"380432b5-86bc-4dd3-a846-d7875d621a36\") " pod="openshift-monitoring/metrics-server-944876b99-29n9f" Oct 30 01:17:57.673826 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.673181 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b098bdf-b302-4954-907b-eadab89054a0-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-8qcgr\" (UID: \"9b098bdf-b302-4954-907b-eadab89054a0\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8qcgr" Oct 30 01:17:57.673826 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.673328 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:57.673826 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.673336 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/380432b5-86bc-4dd3-a846-d7875d621a36-client-ca-bundle\") pod \"metrics-server-944876b99-29n9f\" (UID: \"380432b5-86bc-4dd3-a846-d7875d621a36\") " pod="openshift-monitoring/metrics-server-944876b99-29n9f" Oct 30 01:17:57.673826 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.673344 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b981f97a-f238-4ef6-bbc8-e66cdd90dc90-cert\") pod \"ingress-canary-tk7cl\" (UID: \"b981f97a-f238-4ef6-bbc8-e66cdd90dc90\") " pod="openshift-ingress-canary/ingress-canary-tk7cl" Oct 30 01:17:57.673826 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.673351 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d332bbb4-c699-44e8-96ba-516d8ca03e95-env-overrides\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.673826 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.673361 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9b098bdf-b302-4954-907b-eadab89054a0-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-8qcgr\" (UID: \"9b098bdf-b302-4954-907b-eadab89054a0\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8qcgr" Oct 30 01:17:57.673826 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.673368 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7d834c2a-f008-492e-9aac-362fe7a73672-console-config\") pod \"console-9477b4757-pfqgz\" (UID: \"7d834c2a-f008-492e-9aac-362fe7a73672\") " pod="openshift-console/console-9477b4757-pfqgz" Oct 30 01:17:57.673826 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.673388 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-var-lib-openvswitch\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.673826 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.673487 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d332bbb4-c699-44e8-96ba-516d8ca03e95-ovnkube-config\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.673826 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.673507 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5k2j\" (UniqueName: \"kubernetes.io/projected/4214a36a-fa6a-4290-aadc-2336881f84ba-kube-api-access-t5k2j\") pod \"dns-default-cwzvk\" (UID: \"4214a36a-fa6a-4290-aadc-2336881f84ba\") " pod="openshift-dns/dns-default-cwzvk" Oct 30 01:17:57.673826 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.673523 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e30ba693-6284-485e-95fd-ad639627aa8d-agent-certs\") pod \"konnectivity-agent-9f9b4\" (UID: \"e30ba693-6284-485e-95fd-ad639627aa8d\") " pod="kube-system/konnectivity-agent-9f9b4" Oct 30 01:17:57.673826 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.673562 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/380432b5-86bc-4dd3-a846-d7875d621a36-secret-metrics-server-client-certs\") pod \"metrics-server-944876b99-29n9f\" (UID: \"380432b5-86bc-4dd3-a846-d7875d621a36\") " pod="openshift-monitoring/metrics-server-944876b99-29n9f" Oct 30 01:17:57.673826 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.673563 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/380432b5-86bc-4dd3-a846-d7875d621a36-client-ca-bundle\") pod \"metrics-server-944876b99-29n9f\" (UID: \"380432b5-86bc-4dd3-a846-d7875d621a36\") " pod="openshift-monitoring/metrics-server-944876b99-29n9f" Oct 30 01:17:57.673826 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.673576 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7d834c2a-f008-492e-9aac-362fe7a73672-console-config\") pod \"console-9477b4757-pfqgz\" (UID: \"7d834c2a-f008-492e-9aac-362fe7a73672\") " pod="openshift-console/console-9477b4757-pfqgz" Oct 30 01:17:57.673826 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.673584 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/380432b5-86bc-4dd3-a846-d7875d621a36-audit-log\") pod \"metrics-server-944876b99-29n9f\" (UID: \"380432b5-86bc-4dd3-a846-d7875d621a36\") " pod="openshift-monitoring/metrics-server-944876b99-29n9f" Oct 30 01:17:57.673826 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.673602 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8e7e744e-9806-48ca-9d02-94bb07752f36-console-config\") pod \"console-9bf55565f-w7bkn\" (UID: \"8e7e744e-9806-48ca-9d02-94bb07752f36\") " pod="openshift-console/console-9bf55565f-w7bkn" Oct 30 01:17:57.673826 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.673661 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/380432b5-86bc-4dd3-a846-d7875d621a36-audit-log\") pod \"metrics-server-944876b99-29n9f\" (UID: \"380432b5-86bc-4dd3-a846-d7875d621a36\") " pod="openshift-monitoring/metrics-server-944876b99-29n9f" Oct 30 01:17:57.673826 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.673692 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/380432b5-86bc-4dd3-a846-d7875d621a36-metrics-server-audit-profiles\") pod \"metrics-server-944876b99-29n9f\" (UID: \"380432b5-86bc-4dd3-a846-d7875d621a36\") " pod="openshift-monitoring/metrics-server-944876b99-29n9f" Oct 30 01:17:57.673826 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.673719 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-system-cni-dir\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:17:57.673826 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.673735 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ecb9d3db-582f-41a0-945d-7bd7e4892ff8-serviceca\") pod \"node-ca-lm86p\" (UID: \"ecb9d3db-582f-41a0-945d-7bd7e4892ff8\") " pod="openshift-image-registry/node-ca-lm86p" Oct 30 01:17:57.673826 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.673691 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d332bbb4-c699-44e8-96ba-516d8ca03e95-ovnkube-config\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.673826 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.673761 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a4365b4f-a9d0-41eb-9d89-26591d3e3ed5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2wbfs\" (UID: \"a4365b4f-a9d0-41eb-9d89-26591d3e3ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs" Oct 30 01:17:57.673826 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.673764 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/380432b5-86bc-4dd3-a846-d7875d621a36-secret-metrics-server-client-certs\") pod \"metrics-server-944876b99-29n9f\" (UID: \"380432b5-86bc-4dd3-a846-d7875d621a36\") " pod="openshift-monitoring/metrics-server-944876b99-29n9f" Oct 30 01:17:57.673826 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.673794 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e30ba693-6284-485e-95fd-ad639627aa8d-agent-certs\") pod \"konnectivity-agent-9f9b4\" (UID: \"e30ba693-6284-485e-95fd-ad639627aa8d\") " pod="kube-system/konnectivity-agent-9f9b4" Oct 30 01:17:57.673826 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.673808 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-host-var-lib-cni-bin\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:17:57.675199 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.673938 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/806dfce0-6207-4feb-bee4-ce73cfd4b552-node-exporter-textfile\") pod \"node-exporter-tdj6l\" (UID: \"806dfce0-6207-4feb-bee4-ce73cfd4b552\") " pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:57.675199 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.673972 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-etc-modprobe-d\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:17:57.675199 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.673998 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-config-volume\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:57.675199 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.674003 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/806dfce0-6207-4feb-bee4-ce73cfd4b552-node-exporter-textfile\") pod \"node-exporter-tdj6l\" (UID: \"806dfce0-6207-4feb-bee4-ce73cfd4b552\") " pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:57.675199 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.674022 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-etc-kubernetes\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:17:57.675199 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.674063 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:57.675199 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.674099 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-node-log\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.675199 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.674116 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-multus-cni-dir\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:17:57.675199 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.674138 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/380432b5-86bc-4dd3-a846-d7875d621a36-metrics-server-audit-profiles\") pod \"metrics-server-944876b99-29n9f\" (UID: \"380432b5-86bc-4dd3-a846-d7875d621a36\") " pod="openshift-monitoring/metrics-server-944876b99-29n9f" Oct 30 01:17:57.675199 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.674163 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f10aab9c-d4e6-45b5-a703-146114a31329-kubelet-config\") pod \"global-pull-secret-syncer-dxz45\" (UID: \"f10aab9c-d4e6-45b5-a703-146114a31329\") " pod="kube-system/global-pull-secret-syncer-dxz45" Oct 30 01:17:57.675199 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.674270 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz5xc\" (UniqueName: \"kubernetes.io/projected/8e7e744e-9806-48ca-9d02-94bb07752f36-kube-api-access-zz5xc\") pod \"console-9bf55565f-w7bkn\" (UID: \"8e7e744e-9806-48ca-9d02-94bb07752f36\") " pod="openshift-console/console-9bf55565f-w7bkn" Oct 30 01:17:57.675199 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.674317 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-etc-sysctl-conf\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:17:57.675199 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.674352 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-config-volume\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:57.675199 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.674436 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:57.675199 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.674483 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c819c74a-0f8d-4a5e-9e1b-830196f6b117-hosts-file\") pod \"node-resolver-pf9jv\" (UID: \"c819c74a-0f8d-4a5e-9e1b-830196f6b117\") " pod="openshift-dns/node-resolver-pf9jv" Oct 30 01:17:57.675199 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.674508 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ecb9d3db-582f-41a0-945d-7bd7e4892ff8-serviceca\") pod \"node-ca-lm86p\" (UID: \"ecb9d3db-582f-41a0-945d-7bd7e4892ff8\") " pod="openshift-image-registry/node-ca-lm86p" Oct 30 01:17:57.675199 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.674554 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:57.675199 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.674642 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.675199 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.674678 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-host-run-multus-certs\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:17:57.675199 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.674711 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5wrm\" (UniqueName: \"kubernetes.io/projected/bda81d83-62da-4f40-b668-29e2b0e51f35-kube-api-access-p5wrm\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:17:57.675199 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.674763 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:57.675199 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.674771 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ecb9d3db-582f-41a0-945d-7bd7e4892ff8-host\") pod \"node-ca-lm86p\" (UID: \"ecb9d3db-582f-41a0-945d-7bd7e4892ff8\") " pod="openshift-image-registry/node-ca-lm86p" Oct 30 01:17:57.675199 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.674813 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8e7e744e-9806-48ca-9d02-94bb07752f36-console-oauth-config\") pod \"console-9bf55565f-w7bkn\" (UID: \"8e7e744e-9806-48ca-9d02-94bb07752f36\") " pod="openshift-console/console-9bf55565f-w7bkn" Oct 30 01:17:57.675199 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.674955 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4214a36a-fa6a-4290-aadc-2336881f84ba-tmp-dir\") pod \"dns-default-cwzvk\" (UID: \"4214a36a-fa6a-4290-aadc-2336881f84ba\") " pod="openshift-dns/dns-default-cwzvk" Oct 30 01:17:57.675199 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.674991 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e7e744e-9806-48ca-9d02-94bb07752f36-tmp\") pod \"console-9bf55565f-w7bkn\" (UID: \"8e7e744e-9806-48ca-9d02-94bb07752f36\") " pod="openshift-console/console-9bf55565f-w7bkn" Oct 30 01:17:57.675199 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.675014 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8e7e744e-9806-48ca-9d02-94bb07752f36-console-oauth-config\") pod \"console-9bf55565f-w7bkn\" (UID: \"8e7e744e-9806-48ca-9d02-94bb07752f36\") " pod="openshift-console/console-9bf55565f-w7bkn" Oct 30 01:17:57.675199 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.675026 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 01:17:57.675199 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.675017 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9b098bdf-b302-4954-907b-eadab89054a0-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-8qcgr\" (UID: \"9b098bdf-b302-4954-907b-eadab89054a0\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8qcgr" Oct 30 01:17:57.675199 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.675070 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4214a36a-fa6a-4290-aadc-2336881f84ba-tmp-dir\") pod \"dns-default-cwzvk\" (UID: \"4214a36a-fa6a-4290-aadc-2336881f84ba\") " pod="openshift-dns/dns-default-cwzvk" Oct 30 01:17:57.675199 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.675070 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e7e744e-9806-48ca-9d02-94bb07752f36-tmp\") pod \"console-9bf55565f-w7bkn\" (UID: \"8e7e744e-9806-48ca-9d02-94bb07752f36\") " pod="openshift-console/console-9bf55565f-w7bkn" Oct 30 01:17:57.675199 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.675120 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 01:17:57.675199 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.675135 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c819c74a-0f8d-4a5e-9e1b-830196f6b117-tmp-dir\") pod \"node-resolver-pf9jv\" (UID: \"c819c74a-0f8d-4a5e-9e1b-830196f6b117\") " pod="openshift-dns/node-resolver-pf9jv" Oct 30 01:17:57.675199 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.675163 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7pbf\" (UniqueName: \"kubernetes.io/projected/5618fce7-e2ca-4423-887a-eea73fdd9d01-kube-api-access-n7pbf\") pod \"iptables-alerter-mdwqw\" (UID: \"5618fce7-e2ca-4423-887a-eea73fdd9d01\") " pod="openshift-network-operator/iptables-alerter-mdwqw" Oct 30 01:17:57.675199 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.675182 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-systemd-units\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.675199 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.675186 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c819c74a-0f8d-4a5e-9e1b-830196f6b117-tmp-dir\") pod \"node-resolver-pf9jv\" (UID: \"c819c74a-0f8d-4a5e-9e1b-830196f6b117\") " pod="openshift-dns/node-resolver-pf9jv" Oct 30 01:17:57.675199 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.675198 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-host\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:17:57.675199 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.675193 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9b098bdf-b302-4954-907b-eadab89054a0-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-8qcgr\" (UID: \"9b098bdf-b302-4954-907b-eadab89054a0\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8qcgr" Oct 30 01:17:57.676380 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.675229 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/06f5cece-5273-44d1-90ba-67bf5dc0395b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hft6p\" (UID: \"06f5cece-5273-44d1-90ba-67bf5dc0395b\") " pod="openshift-insights/insights-runtime-extractor-hft6p" Oct 30 01:17:57.676380 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.675245 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w5t7\" (UniqueName: \"kubernetes.io/projected/d332bbb4-c699-44e8-96ba-516d8ca03e95-kube-api-access-8w5t7\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.676380 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.675303 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/806dfce0-6207-4feb-bee4-ce73cfd4b552-metrics-client-ca\") pod \"node-exporter-tdj6l\" (UID: \"806dfce0-6207-4feb-bee4-ce73cfd4b552\") " pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:57.676380 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.675333 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db13cab2-9b76-44b5-8577-9f8f0d3f292e-service-ca\") pod \"console-7b6fbb6f6f-9z4ws\" (UID: \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\") " pod="openshift-console/console-7b6fbb6f6f-9z4ws" Oct 30 01:17:57.676380 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.675358 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d205185b-fc45-4a76-9719-20b00107abf1-config-out\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:57.676380 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.675372 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/06f5cece-5273-44d1-90ba-67bf5dc0395b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hft6p\" (UID: \"06f5cece-5273-44d1-90ba-67bf5dc0395b\") " pod="openshift-insights/insights-runtime-extractor-hft6p" Oct 30 01:17:57.676380 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.675384 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d834c2a-f008-492e-9aac-362fe7a73672-service-ca\") pod \"console-9477b4757-pfqgz\" (UID: \"7d834c2a-f008-492e-9aac-362fe7a73672\") " pod="openshift-console/console-9477b4757-pfqgz" Oct 30 01:17:57.676380 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.675412 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2ec1edc-b911-4537-ac00-f708ef6c9062-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xfvmk\" (UID: \"a2ec1edc-b911-4537-ac00-f708ef6c9062\") " pod="openshift-multus/multus-additional-cni-plugins-xfvmk" Oct 30 01:17:57.676380 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.675425 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d205185b-fc45-4a76-9719-20b00107abf1-config-out\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:57.676380 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.675440 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-host-var-lib-cni-multus\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:17:57.676380 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.675475 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nb6l\" (UniqueName: \"kubernetes.io/projected/9b098bdf-b302-4954-907b-eadab89054a0-kube-api-access-5nb6l\") pod \"prometheus-operator-9cd89b74c-8qcgr\" (UID: \"9b098bdf-b302-4954-907b-eadab89054a0\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8qcgr" Oct 30 01:17:57.676380 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.675504 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/806dfce0-6207-4feb-bee4-ce73cfd4b552-metrics-client-ca\") pod \"node-exporter-tdj6l\" (UID: \"806dfce0-6207-4feb-bee4-ce73cfd4b552\") " pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:57.676380 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.675515 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db13cab2-9b76-44b5-8577-9f8f0d3f292e-service-ca\") pod \"console-7b6fbb6f6f-9z4ws\" (UID: \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\") " pod="openshift-console/console-7b6fbb6f6f-9z4ws" Oct 30 01:17:57.676380 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.675694 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d834c2a-f008-492e-9aac-362fe7a73672-service-ca\") pod \"console-9477b4757-pfqgz\" (UID: \"7d834c2a-f008-492e-9aac-362fe7a73672\") " pod="openshift-console/console-9477b4757-pfqgz" Oct 30 01:17:57.679886 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.679863 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d834c2a-f008-492e-9aac-362fe7a73672-trusted-ca-bundle\") pod \"console-9477b4757-pfqgz\" (UID: \"7d834c2a-f008-492e-9aac-362fe7a73672\") " pod="openshift-console/console-9477b4757-pfqgz" Oct 30 01:17:57.679963 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.679865 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d205185b-fc45-4a76-9719-20b00107abf1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:57.682665 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.682645 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db13cab2-9b76-44b5-8577-9f8f0d3f292e-trusted-ca-bundle\") pod \"console-7b6fbb6f6f-9z4ws\" (UID: \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\") " pod="openshift-console/console-7b6fbb6f6f-9z4ws" Oct 30 01:17:57.693798 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.693775 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qbvg\" (UniqueName: \"kubernetes.io/projected/a2ec1edc-b911-4537-ac00-f708ef6c9062-kube-api-access-8qbvg\") pod \"multus-additional-cni-plugins-xfvmk\" (UID: \"a2ec1edc-b911-4537-ac00-f708ef6c9062\") " pod="openshift-multus/multus-additional-cni-plugins-xfvmk" Oct 30 01:17:57.695890 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.695869 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgt8r\" (UniqueName: \"kubernetes.io/projected/806dfce0-6207-4feb-bee4-ce73cfd4b552-kube-api-access-bgt8r\") pod \"node-exporter-tdj6l\" (UID: \"806dfce0-6207-4feb-bee4-ce73cfd4b552\") " pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:57.695967 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.695938 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f67fc\" (UniqueName: \"kubernetes.io/projected/a4365b4f-a9d0-41eb-9d89-26591d3e3ed5-kube-api-access-f67fc\") pod \"aws-ebs-csi-driver-node-2wbfs\" (UID: \"a4365b4f-a9d0-41eb-9d89-26591d3e3ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs" Oct 30 01:17:57.696615 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.696593 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjqdw\" (UniqueName: \"kubernetes.io/projected/380432b5-86bc-4dd3-a846-d7875d621a36-kube-api-access-wjqdw\") pod \"metrics-server-944876b99-29n9f\" (UID: \"380432b5-86bc-4dd3-a846-d7875d621a36\") " pod="openshift-monitoring/metrics-server-944876b99-29n9f" Oct 30 01:17:57.696671 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.696658 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzhgj\" (UniqueName: \"kubernetes.io/projected/b981f97a-f238-4ef6-bbc8-e66cdd90dc90-kube-api-access-hzhgj\") pod \"ingress-canary-tk7cl\" (UID: \"b981f97a-f238-4ef6-bbc8-e66cdd90dc90\") " pod="openshift-ingress-canary/ingress-canary-tk7cl" Oct 30 01:17:57.697908 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.697893 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd48f\" (UniqueName: \"kubernetes.io/projected/06f5cece-5273-44d1-90ba-67bf5dc0395b-kube-api-access-jd48f\") pod \"insights-runtime-extractor-hft6p\" (UID: \"06f5cece-5273-44d1-90ba-67bf5dc0395b\") " pod="openshift-insights/insights-runtime-extractor-hft6p" Oct 30 01:17:57.701003 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.700984 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd9jh\" (UniqueName: \"kubernetes.io/projected/7d834c2a-f008-492e-9aac-362fe7a73672-kube-api-access-sd9jh\") pod \"console-9477b4757-pfqgz\" (UID: \"7d834c2a-f008-492e-9aac-362fe7a73672\") " pod="openshift-console/console-9477b4757-pfqgz" Oct 30 01:17:57.701082 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.701065 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89k2s\" (UniqueName: \"kubernetes.io/projected/f4602ac2-652b-4026-9a0d-d72162bb3258-kube-api-access-89k2s\") pod \"network-check-target-bbhx5\" (UID: \"f4602ac2-652b-4026-9a0d-d72162bb3258\") " pod="openshift-network-diagnostics/network-check-target-bbhx5" Oct 30 01:17:57.701516 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.701499 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w5t7\" (UniqueName: \"kubernetes.io/projected/d332bbb4-c699-44e8-96ba-516d8ca03e95-kube-api-access-8w5t7\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.702750 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.702732 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5k2j\" (UniqueName: \"kubernetes.io/projected/4214a36a-fa6a-4290-aadc-2336881f84ba-kube-api-access-t5k2j\") pod \"dns-default-cwzvk\" (UID: \"4214a36a-fa6a-4290-aadc-2336881f84ba\") " pod="openshift-dns/dns-default-cwzvk" Oct 30 01:17:57.702942 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.702925 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rmpm\" (UniqueName: \"kubernetes.io/projected/94864d90-1dd8-472e-a414-29c6b6a4f8f6-kube-api-access-5rmpm\") pod \"network-metrics-daemon-xj8mh\" (UID: \"94864d90-1dd8-472e-a414-29c6b6a4f8f6\") " pod="openshift-multus/network-metrics-daemon-xj8mh" Oct 30 01:17:57.703283 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.703263 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dhns\" (UniqueName: \"kubernetes.io/projected/c819c74a-0f8d-4a5e-9e1b-830196f6b117-kube-api-access-4dhns\") pod \"node-resolver-pf9jv\" (UID: \"c819c74a-0f8d-4a5e-9e1b-830196f6b117\") " pod="openshift-dns/node-resolver-pf9jv" Oct 30 01:17:57.707393 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.707371 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdvvv\" (UniqueName: \"kubernetes.io/projected/ecb9d3db-582f-41a0-945d-7bd7e4892ff8-kube-api-access-kdvvv\") pod \"node-ca-lm86p\" (UID: \"ecb9d3db-582f-41a0-945d-7bd7e4892ff8\") " pod="openshift-image-registry/node-ca-lm86p" Oct 30 01:17:57.707711 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.707686 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5wrm\" (UniqueName: \"kubernetes.io/projected/bda81d83-62da-4f40-b668-29e2b0e51f35-kube-api-access-p5wrm\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:17:57.707817 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.707801 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nb6l\" (UniqueName: \"kubernetes.io/projected/9b098bdf-b302-4954-907b-eadab89054a0-kube-api-access-5nb6l\") pod \"prometheus-operator-9cd89b74c-8qcgr\" (UID: \"9b098bdf-b302-4954-907b-eadab89054a0\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8qcgr" Oct 30 01:17:57.707817 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.707809 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz5xc\" (UniqueName: \"kubernetes.io/projected/8e7e744e-9806-48ca-9d02-94bb07752f36-kube-api-access-zz5xc\") pod \"console-9bf55565f-w7bkn\" (UID: \"8e7e744e-9806-48ca-9d02-94bb07752f36\") " pod="openshift-console/console-9bf55565f-w7bkn" Oct 30 01:17:57.707907 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.707834 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7frp\" (UniqueName: \"kubernetes.io/projected/db13cab2-9b76-44b5-8577-9f8f0d3f292e-kube-api-access-l7frp\") pod \"console-7b6fbb6f6f-9z4ws\" (UID: \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\") " pod="openshift-console/console-7b6fbb6f6f-9z4ws" Oct 30 01:17:57.708097 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.708081 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcd88\" (UniqueName: \"kubernetes.io/projected/d205185b-fc45-4a76-9719-20b00107abf1-kube-api-access-mcd88\") pod \"alertmanager-main-0\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:17:57.709796 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.709780 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmptt\" (UniqueName: \"kubernetes.io/projected/e09d6f2e-7cd5-47c3-b18b-482726067e90-kube-api-access-nmptt\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:17:57.710050 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.710030 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7pbf\" (UniqueName: \"kubernetes.io/projected/5618fce7-e2ca-4423-887a-eea73fdd9d01-kube-api-access-n7pbf\") pod \"iptables-alerter-mdwqw\" (UID: \"5618fce7-e2ca-4423-887a-eea73fdd9d01\") " pod="openshift-network-operator/iptables-alerter-mdwqw" Oct 30 01:17:57.720721 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.720701 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9bf55565f-w7bkn"] Oct 30 01:17:57.776610 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.776509 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-host-var-lib-kubelet\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:17:57.776610 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.776564 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a4365b4f-a9d0-41eb-9d89-26591d3e3ed5-device-dir\") pod \"aws-ebs-csi-driver-node-2wbfs\" (UID: \"a4365b4f-a9d0-41eb-9d89-26591d3e3ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs" Oct 30 01:17:57.776610 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.776608 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-run-openvswitch\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.776864 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.776626 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-host-cni-bin\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.776864 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.776637 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a4365b4f-a9d0-41eb-9d89-26591d3e3ed5-device-dir\") pod \"aws-ebs-csi-driver-node-2wbfs\" (UID: \"a4365b4f-a9d0-41eb-9d89-26591d3e3ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs" Oct 30 01:17:57.776864 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.776645 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-host-var-lib-kubelet\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:17:57.776864 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.776643 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-host-slash\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.776864 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.776685 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-host-slash\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.776864 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.776722 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-host-cni-bin\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.776864 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.776722 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-run-openvswitch\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.776864 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.776736 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a4365b4f-a9d0-41eb-9d89-26591d3e3ed5-socket-dir\") pod \"aws-ebs-csi-driver-node-2wbfs\" (UID: \"a4365b4f-a9d0-41eb-9d89-26591d3e3ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs" Oct 30 01:17:57.776864 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.776761 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a4365b4f-a9d0-41eb-9d89-26591d3e3ed5-etc-selinux\") pod \"aws-ebs-csi-driver-node-2wbfs\" (UID: \"a4365b4f-a9d0-41eb-9d89-26591d3e3ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs" Oct 30 01:17:57.776864 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.776778 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/806dfce0-6207-4feb-bee4-ce73cfd4b552-sys\") pod \"node-exporter-tdj6l\" (UID: \"806dfce0-6207-4feb-bee4-ce73cfd4b552\") " pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:57.776864 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.776794 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-multus-socket-dir-parent\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:17:57.776864 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.776819 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-etc-systemd\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:17:57.776864 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.776823 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a4365b4f-a9d0-41eb-9d89-26591d3e3ed5-socket-dir\") pod \"aws-ebs-csi-driver-node-2wbfs\" (UID: \"a4365b4f-a9d0-41eb-9d89-26591d3e3ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs" Oct 30 01:17:57.776864 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.776833 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/806dfce0-6207-4feb-bee4-ce73cfd4b552-sys\") pod \"node-exporter-tdj6l\" (UID: \"806dfce0-6207-4feb-bee4-ce73cfd4b552\") " pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:57.776864 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.776839 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a4365b4f-a9d0-41eb-9d89-26591d3e3ed5-etc-selinux\") pod \"aws-ebs-csi-driver-node-2wbfs\" (UID: \"a4365b4f-a9d0-41eb-9d89-26591d3e3ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs" Oct 30 01:17:57.776864 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.776844 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-etc-sysctl-d\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:17:57.777416 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.776884 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-multus-socket-dir-parent\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:17:57.777416 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.776903 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-var-lib-kubelet\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:17:57.777416 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.776923 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-etc-systemd\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:17:57.777416 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.776956 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-etc-sysctl-d\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:17:57.777416 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.776972 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-run\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:17:57.777416 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.776998 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-etc-openvswitch\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.777416 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777012 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-run-ovn\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.777416 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777028 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-host-run-k8s-cni-cncf-io\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:17:57.777416 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777023 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-run\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:17:57.777416 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777046 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-host-kubelet\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.777416 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777062 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-log-socket\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.777416 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777084 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-etc-openvswitch\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.777416 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777090 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-var-lib-kubelet\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:17:57.777416 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777091 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-host-run-k8s-cni-cncf-io\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:17:57.777416 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777092 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-log-socket\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.777416 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777102 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-host-kubelet\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.777416 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777131 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-run-ovn\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.777416 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777134 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/806dfce0-6207-4feb-bee4-ce73cfd4b552-node-exporter-wtmp\") pod \"node-exporter-tdj6l\" (UID: \"806dfce0-6207-4feb-bee4-ce73cfd4b552\") " pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:57.777416 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777179 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/806dfce0-6207-4feb-bee4-ce73cfd4b552-node-exporter-wtmp\") pod \"node-exporter-tdj6l\" (UID: \"806dfce0-6207-4feb-bee4-ce73cfd4b552\") " pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:57.777416 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777192 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-host-run-netns\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:17:57.777416 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777220 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-etc-kubernetes\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:17:57.777416 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777246 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-host-run-netns\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.777416 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777262 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2ec1edc-b911-4537-ac00-f708ef6c9062-cnibin\") pod \"multus-additional-cni-plugins-xfvmk\" (UID: \"a2ec1edc-b911-4537-ac00-f708ef6c9062\") " pod="openshift-multus/multus-additional-cni-plugins-xfvmk" Oct 30 01:17:57.777416 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777278 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-multus-conf-dir\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:17:57.777416 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777284 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-etc-kubernetes\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:17:57.777416 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777221 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-host-run-netns\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:17:57.777416 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777300 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-etc-sysconfig\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:17:57.777416 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777322 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-host-run-netns\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.777416 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777328 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2ec1edc-b911-4537-ac00-f708ef6c9062-cnibin\") pod \"multus-additional-cni-plugins-xfvmk\" (UID: \"a2ec1edc-b911-4537-ac00-f708ef6c9062\") " pod="openshift-multus/multus-additional-cni-plugins-xfvmk" Oct 30 01:17:57.777416 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777333 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-multus-conf-dir\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:17:57.777416 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777326 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/806dfce0-6207-4feb-bee4-ce73cfd4b552-root\") pod \"node-exporter-tdj6l\" (UID: \"806dfce0-6207-4feb-bee4-ce73cfd4b552\") " pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:57.777416 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777361 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/806dfce0-6207-4feb-bee4-ce73cfd4b552-root\") pod \"node-exporter-tdj6l\" (UID: \"806dfce0-6207-4feb-bee4-ce73cfd4b552\") " pod="openshift-monitoring/node-exporter-tdj6l" Oct 30 01:17:57.777416 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777363 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-var-lib-openvswitch\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.777416 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777373 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-etc-sysconfig\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:17:57.777416 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777382 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-var-lib-openvswitch\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.777416 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777428 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-system-cni-dir\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777470 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a4365b4f-a9d0-41eb-9d89-26591d3e3ed5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2wbfs\" (UID: \"a4365b4f-a9d0-41eb-9d89-26591d3e3ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777489 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-host-var-lib-cni-bin\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777487 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-system-cni-dir\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777508 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-etc-modprobe-d\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777523 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-etc-kubernetes\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777553 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a4365b4f-a9d0-41eb-9d89-26591d3e3ed5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2wbfs\" (UID: \"a4365b4f-a9d0-41eb-9d89-26591d3e3ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777566 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-node-log\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777586 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-multus-cni-dir\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777593 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-etc-kubernetes\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777567 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-host-var-lib-cni-bin\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777606 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f10aab9c-d4e6-45b5-a703-146114a31329-kubelet-config\") pod \"global-pull-secret-syncer-dxz45\" (UID: \"f10aab9c-d4e6-45b5-a703-146114a31329\") " pod="kube-system/global-pull-secret-syncer-dxz45" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777649 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-etc-modprobe-d\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777655 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-multus-cni-dir\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777703 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-etc-sysctl-conf\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777727 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c819c74a-0f8d-4a5e-9e1b-830196f6b117-hosts-file\") pod \"node-resolver-pf9jv\" (UID: \"c819c74a-0f8d-4a5e-9e1b-830196f6b117\") " pod="openshift-dns/node-resolver-pf9jv" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777746 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f10aab9c-d4e6-45b5-a703-146114a31329-kubelet-config\") pod \"global-pull-secret-syncer-dxz45\" (UID: \"f10aab9c-d4e6-45b5-a703-146114a31329\") " pod="kube-system/global-pull-secret-syncer-dxz45" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777751 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777792 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-host-run-multus-certs\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777799 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c819c74a-0f8d-4a5e-9e1b-830196f6b117-hosts-file\") pod \"node-resolver-pf9jv\" (UID: \"c819c74a-0f8d-4a5e-9e1b-830196f6b117\") " pod="openshift-dns/node-resolver-pf9jv" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777831 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-etc-sysctl-conf\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777839 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-host-run-multus-certs\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777836 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777851 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ecb9d3db-582f-41a0-945d-7bd7e4892ff8-host\") pod \"node-ca-lm86p\" (UID: \"ecb9d3db-582f-41a0-945d-7bd7e4892ff8\") " pod="openshift-image-registry/node-ca-lm86p" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777891 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-systemd-units\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777906 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-host\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777930 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ecb9d3db-582f-41a0-945d-7bd7e4892ff8-host\") pod \"node-ca-lm86p\" (UID: \"ecb9d3db-582f-41a0-945d-7bd7e4892ff8\") " pod="openshift-image-registry/node-ca-lm86p" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777926 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2ec1edc-b911-4537-ac00-f708ef6c9062-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xfvmk\" (UID: \"a2ec1edc-b911-4537-ac00-f708ef6c9062\") " pod="openshift-multus/multus-additional-cni-plugins-xfvmk" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777965 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-host-var-lib-cni-multus\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777964 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2ec1edc-b911-4537-ac00-f708ef6c9062-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xfvmk\" (UID: \"a2ec1edc-b911-4537-ac00-f708ef6c9062\") " pod="openshift-multus/multus-additional-cni-plugins-xfvmk" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777999 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-host-var-lib-cni-multus\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.778013 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-systemd-units\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.778034 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-host\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.777592 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-node-log\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.778061 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-sys\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.778087 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/06f5cece-5273-44d1-90ba-67bf5dc0395b-crio-socket\") pod \"insights-runtime-extractor-hft6p\" (UID: \"06f5cece-5273-44d1-90ba-67bf5dc0395b\") " pod="openshift-insights/insights-runtime-extractor-hft6p" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.778103 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2ec1edc-b911-4537-ac00-f708ef6c9062-system-cni-dir\") pod \"multus-additional-cni-plugins-xfvmk\" (UID: \"a2ec1edc-b911-4537-ac00-f708ef6c9062\") " pod="openshift-multus/multus-additional-cni-plugins-xfvmk" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.778120 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-hostroot\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.778136 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a4365b4f-a9d0-41eb-9d89-26591d3e3ed5-sys-fs\") pod \"aws-ebs-csi-driver-node-2wbfs\" (UID: \"a4365b4f-a9d0-41eb-9d89-26591d3e3ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.778165 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f10aab9c-d4e6-45b5-a703-146114a31329-dbus\") pod \"global-pull-secret-syncer-dxz45\" (UID: \"f10aab9c-d4e6-45b5-a703-146114a31329\") " pod="kube-system/global-pull-secret-syncer-dxz45" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.778158 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/06f5cece-5273-44d1-90ba-67bf5dc0395b-crio-socket\") pod \"insights-runtime-extractor-hft6p\" (UID: \"06f5cece-5273-44d1-90ba-67bf5dc0395b\") " pod="openshift-insights/insights-runtime-extractor-hft6p" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.778193 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-hostroot\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.778199 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2ec1edc-b911-4537-ac00-f708ef6c9062-system-cni-dir\") pod \"multus-additional-cni-plugins-xfvmk\" (UID: \"a2ec1edc-b911-4537-ac00-f708ef6c9062\") " pod="openshift-multus/multus-additional-cni-plugins-xfvmk" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.778226 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f10aab9c-d4e6-45b5-a703-146114a31329-dbus\") pod \"global-pull-secret-syncer-dxz45\" (UID: \"f10aab9c-d4e6-45b5-a703-146114a31329\") " pod="kube-system/global-pull-secret-syncer-dxz45" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.778230 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-lib-modules\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.778230 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-sys\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.778231 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a4365b4f-a9d0-41eb-9d89-26591d3e3ed5-sys-fs\") pod \"aws-ebs-csi-driver-node-2wbfs\" (UID: \"a4365b4f-a9d0-41eb-9d89-26591d3e3ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.778253 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a4365b4f-a9d0-41eb-9d89-26591d3e3ed5-registration-dir\") pod \"aws-ebs-csi-driver-node-2wbfs\" (UID: \"a4365b4f-a9d0-41eb-9d89-26591d3e3ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.778288 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-run-systemd\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.778318 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-host-run-ovn-kubernetes\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.778325 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-run-systemd\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.778332 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-cnibin\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.778336 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e09d6f2e-7cd5-47c3-b18b-482726067e90-lib-modules\") pod \"tuned-pkkfl\" (UID: \"e09d6f2e-7cd5-47c3-b18b-482726067e90\") " pod="openshift-cluster-node-tuning-operator/tuned-pkkfl" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.778335 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a4365b4f-a9d0-41eb-9d89-26591d3e3ed5-registration-dir\") pod \"aws-ebs-csi-driver-node-2wbfs\" (UID: \"a4365b4f-a9d0-41eb-9d89-26591d3e3ed5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2wbfs" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.778355 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5618fce7-e2ca-4423-887a-eea73fdd9d01-host-slash\") pod \"iptables-alerter-mdwqw\" (UID: \"5618fce7-e2ca-4423-887a-eea73fdd9d01\") " pod="openshift-network-operator/iptables-alerter-mdwqw" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.778353 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-host-run-ovn-kubernetes\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.778385 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-host-cni-netd\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.778393 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bda81d83-62da-4f40-b668-29e2b0e51f35-cnibin\") pod \"multus-pgk7v\" (UID: \"bda81d83-62da-4f40-b668-29e2b0e51f35\") " pod="openshift-multus/multus-pgk7v" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.778399 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5618fce7-e2ca-4423-887a-eea73fdd9d01-host-slash\") pod \"iptables-alerter-mdwqw\" (UID: \"5618fce7-e2ca-4423-887a-eea73fdd9d01\") " pod="openshift-network-operator/iptables-alerter-mdwqw" Oct 30 01:17:57.778770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:57.778476 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d332bbb4-c699-44e8-96ba-516d8ca03e95-host-cni-netd\") pod \"ovnkube-node-2vsns\" (UID: \"d332bbb4-c699-44e8-96ba-516d8ca03e95\") " pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:17:59.283251 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:17:59.283217 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-cwzvk" Oct 30 01:18:01.732041 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:01.732011 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:18:01.916747 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:01.916707 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:18:02.366984 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:02.366951 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9477b4757-pfqgz" Oct 30 01:18:02.619070 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:02.618988 6221 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 01:18:03.062964 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:03.062926 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9477b4757-pfqgz" Oct 30 01:18:03.658435 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:03.658382 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bbhx5" Oct 30 01:18:04.644339 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:04.644310 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9f9b4" Oct 30 01:18:05.329108 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:05.329073 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9bf55565f-w7bkn" Oct 30 01:18:05.765673 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:05.765609 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-944876b99-29n9f" Oct 30 01:18:06.722555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:06.722517 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2vsns" Oct 30 01:18:12.371936 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:12.371858 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9477b4757-pfqgz" Oct 30 01:18:12.375526 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:12.375501 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9477b4757-pfqgz" Oct 30 01:18:21.492092 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:18:21.492057 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:18:22.575007 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:22.574968 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7b6fbb6f6f-9z4ws" podUID="db13cab2-9b76-44b5-8577-9f8f0d3f292e" containerName="console" containerID="cri-o://2240a9d5917c8ae0ac143c8af0a3020dba0bc4a18ace2943ce2ebba50d98a22f" gracePeriod=15 Oct 30 01:18:22.808634 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:22.808612 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b6fbb6f6f-9z4ws" Oct 30 01:18:22.868719 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:22.868658 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db13cab2-9b76-44b5-8577-9f8f0d3f292e-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db13cab2-9b76-44b5-8577-9f8f0d3f292e-oauth-serving-cert\") pod \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\" (UID: \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\") " Oct 30 01:18:22.868719 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:22.868710 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db13cab2-9b76-44b5-8577-9f8f0d3f292e-tmp\" (UniqueName: \"kubernetes.io/empty-dir/db13cab2-9b76-44b5-8577-9f8f0d3f292e-tmp\") pod \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\" (UID: \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\") " Oct 30 01:18:22.868905 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:22.868748 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db13cab2-9b76-44b5-8577-9f8f0d3f292e-kube-api-access-l7frp\" (UniqueName: \"kubernetes.io/projected/db13cab2-9b76-44b5-8577-9f8f0d3f292e-kube-api-access-l7frp\") pod \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\" (UID: \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\") " Oct 30 01:18:22.868905 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:22.868791 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db13cab2-9b76-44b5-8577-9f8f0d3f292e-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db13cab2-9b76-44b5-8577-9f8f0d3f292e-console-serving-cert\") pod \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\" (UID: \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\") " Oct 30 01:18:22.868905 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:22.868822 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db13cab2-9b76-44b5-8577-9f8f0d3f292e-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db13cab2-9b76-44b5-8577-9f8f0d3f292e-trusted-ca-bundle\") pod \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\" (UID: \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\") " Oct 30 01:18:22.868905 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:22.868856 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db13cab2-9b76-44b5-8577-9f8f0d3f292e-console-config\" (UniqueName: \"kubernetes.io/configmap/db13cab2-9b76-44b5-8577-9f8f0d3f292e-console-config\") pod \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\" (UID: \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\") " Oct 30 01:18:22.868905 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:22.868895 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db13cab2-9b76-44b5-8577-9f8f0d3f292e-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db13cab2-9b76-44b5-8577-9f8f0d3f292e-console-oauth-config\") pod \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\" (UID: \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\") " Oct 30 01:18:22.869152 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:22.868933 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db13cab2-9b76-44b5-8577-9f8f0d3f292e-service-ca\" (UniqueName: \"kubernetes.io/configmap/db13cab2-9b76-44b5-8577-9f8f0d3f292e-service-ca\") pod \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\" (UID: \"db13cab2-9b76-44b5-8577-9f8f0d3f292e\") " Oct 30 01:18:22.869152 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:22.869060 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db13cab2-9b76-44b5-8577-9f8f0d3f292e-tmp" pod "db13cab2-9b76-44b5-8577-9f8f0d3f292e" (UID: "db13cab2-9b76-44b5-8577-9f8f0d3f292e"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:18:22.869267 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:22.869067 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db13cab2-9b76-44b5-8577-9f8f0d3f292e-oauth-serving-cert" pod "db13cab2-9b76-44b5-8577-9f8f0d3f292e" (UID: "db13cab2-9b76-44b5-8577-9f8f0d3f292e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:18:22.869267 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:22.869173 6221 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db13cab2-9b76-44b5-8577-9f8f0d3f292e-tmp\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:18:22.869422 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:22.869398 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db13cab2-9b76-44b5-8577-9f8f0d3f292e-service-ca" pod "db13cab2-9b76-44b5-8577-9f8f0d3f292e" (UID: "db13cab2-9b76-44b5-8577-9f8f0d3f292e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:18:22.869422 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:22.869407 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db13cab2-9b76-44b5-8577-9f8f0d3f292e-console-config" pod "db13cab2-9b76-44b5-8577-9f8f0d3f292e" (UID: "db13cab2-9b76-44b5-8577-9f8f0d3f292e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:18:22.869604 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:22.869582 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db13cab2-9b76-44b5-8577-9f8f0d3f292e-trusted-ca-bundle" pod "db13cab2-9b76-44b5-8577-9f8f0d3f292e" (UID: "db13cab2-9b76-44b5-8577-9f8f0d3f292e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:18:22.871178 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:22.871154 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db13cab2-9b76-44b5-8577-9f8f0d3f292e-console-serving-cert" pod "db13cab2-9b76-44b5-8577-9f8f0d3f292e" (UID: "db13cab2-9b76-44b5-8577-9f8f0d3f292e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:18:22.871307 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:22.871282 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db13cab2-9b76-44b5-8577-9f8f0d3f292e-kube-api-access-l7frp" pod "db13cab2-9b76-44b5-8577-9f8f0d3f292e" (UID: "db13cab2-9b76-44b5-8577-9f8f0d3f292e"). InnerVolumeSpecName "kube-api-access-l7frp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:18:22.871458 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:22.871444 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db13cab2-9b76-44b5-8577-9f8f0d3f292e-console-oauth-config" pod "db13cab2-9b76-44b5-8577-9f8f0d3f292e" (UID: "db13cab2-9b76-44b5-8577-9f8f0d3f292e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:18:22.969523 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:22.969495 6221 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db13cab2-9b76-44b5-8577-9f8f0d3f292e-trusted-ca-bundle\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:18:22.969523 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:22.969523 6221 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db13cab2-9b76-44b5-8577-9f8f0d3f292e-console-config\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:18:22.969731 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:22.969558 6221 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db13cab2-9b76-44b5-8577-9f8f0d3f292e-console-oauth-config\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:18:22.969731 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:22.969571 6221 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db13cab2-9b76-44b5-8577-9f8f0d3f292e-service-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:18:22.969731 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:22.969583 6221 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db13cab2-9b76-44b5-8577-9f8f0d3f292e-oauth-serving-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:18:22.969731 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:22.969595 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7frp\" (UniqueName: \"kubernetes.io/projected/db13cab2-9b76-44b5-8577-9f8f0d3f292e-kube-api-access-l7frp\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:18:22.969731 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:22.969609 6221 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db13cab2-9b76-44b5-8577-9f8f0d3f292e-console-serving-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:18:23.620119 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:23.620083 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-9bf55565f-w7bkn" podUID="8e7e744e-9806-48ca-9d02-94bb07752f36" containerName="console" containerID="cri-o://fc1bd0a7e19c1ff001def32af6a08009019658216ffa26169a4abeb435cdfeb0" gracePeriod=15 Oct 30 01:18:23.688519 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:23.688483 6221 generic.go:356] "Generic (PLEG): container finished" podID="db13cab2-9b76-44b5-8577-9f8f0d3f292e" containerID="2240a9d5917c8ae0ac143c8af0a3020dba0bc4a18ace2943ce2ebba50d98a22f" exitCode=0 Oct 30 01:18:23.688653 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:23.688550 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b6fbb6f6f-9z4ws" Oct 30 01:18:23.688653 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:23.688569 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b6fbb6f6f-9z4ws" event={"ID":"db13cab2-9b76-44b5-8577-9f8f0d3f292e","Type":"ContainerDied","Data":"2240a9d5917c8ae0ac143c8af0a3020dba0bc4a18ace2943ce2ebba50d98a22f"} Oct 30 01:18:23.688653 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:23.688622 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b6fbb6f6f-9z4ws" event={"ID":"db13cab2-9b76-44b5-8577-9f8f0d3f292e","Type":"ContainerDied","Data":"223f329baedfd1629bc7e494bc60ac6c4b654bcbc118acaff1ad96b96e5fc52a"} Oct 30 01:18:23.688653 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:23.688643 6221 scope.go:122] "RemoveContainer" containerID="2240a9d5917c8ae0ac143c8af0a3020dba0bc4a18ace2943ce2ebba50d98a22f" Oct 30 01:18:23.730570 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:23.730547 6221 scope.go:122] "RemoveContainer" containerID="2240a9d5917c8ae0ac143c8af0a3020dba0bc4a18ace2943ce2ebba50d98a22f" Oct 30 01:18:23.730852 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:18:23.730833 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2240a9d5917c8ae0ac143c8af0a3020dba0bc4a18ace2943ce2ebba50d98a22f\": container with ID starting with 2240a9d5917c8ae0ac143c8af0a3020dba0bc4a18ace2943ce2ebba50d98a22f not found: ID does not exist" containerID="2240a9d5917c8ae0ac143c8af0a3020dba0bc4a18ace2943ce2ebba50d98a22f" Oct 30 01:18:23.730927 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:23.730863 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2240a9d5917c8ae0ac143c8af0a3020dba0bc4a18ace2943ce2ebba50d98a22f"} err="failed to get container status \"2240a9d5917c8ae0ac143c8af0a3020dba0bc4a18ace2943ce2ebba50d98a22f\": rpc error: code = NotFound desc = could not find container \"2240a9d5917c8ae0ac143c8af0a3020dba0bc4a18ace2943ce2ebba50d98a22f\": container with ID starting with 2240a9d5917c8ae0ac143c8af0a3020dba0bc4a18ace2943ce2ebba50d98a22f not found: ID does not exist" Oct 30 01:18:23.739022 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:23.738997 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b6fbb6f6f-9z4ws"] Oct 30 01:18:23.751055 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:23.751032 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7b6fbb6f6f-9z4ws"] Oct 30 01:18:23.867656 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:23.867631 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9bf55565f-w7bkn" Oct 30 01:18:23.876949 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:23.876905 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e7e744e-9806-48ca-9d02-94bb07752f36-service-ca\" (UniqueName: \"kubernetes.io/configmap/8e7e744e-9806-48ca-9d02-94bb07752f36-service-ca\") pod \"8e7e744e-9806-48ca-9d02-94bb07752f36\" (UID: \"8e7e744e-9806-48ca-9d02-94bb07752f36\") " Oct 30 01:18:23.876949 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:23.876945 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e7e744e-9806-48ca-9d02-94bb07752f36-tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e7e744e-9806-48ca-9d02-94bb07752f36-tmp\") pod \"8e7e744e-9806-48ca-9d02-94bb07752f36\" (UID: \"8e7e744e-9806-48ca-9d02-94bb07752f36\") " Oct 30 01:18:23.877056 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:23.876973 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e7e744e-9806-48ca-9d02-94bb07752f36-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8e7e744e-9806-48ca-9d02-94bb07752f36-oauth-serving-cert\") pod \"8e7e744e-9806-48ca-9d02-94bb07752f36\" (UID: \"8e7e744e-9806-48ca-9d02-94bb07752f36\") " Oct 30 01:18:23.877056 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:23.877023 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e7e744e-9806-48ca-9d02-94bb07752f36-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e7e744e-9806-48ca-9d02-94bb07752f36-console-serving-cert\") pod \"8e7e744e-9806-48ca-9d02-94bb07752f36\" (UID: \"8e7e744e-9806-48ca-9d02-94bb07752f36\") " Oct 30 01:18:23.877158 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:23.877131 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e7e744e-9806-48ca-9d02-94bb07752f36-kube-api-access-zz5xc\" (UniqueName: \"kubernetes.io/projected/8e7e744e-9806-48ca-9d02-94bb07752f36-kube-api-access-zz5xc\") pod \"8e7e744e-9806-48ca-9d02-94bb07752f36\" (UID: \"8e7e744e-9806-48ca-9d02-94bb07752f36\") " Oct 30 01:18:23.877214 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:23.877182 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e7e744e-9806-48ca-9d02-94bb07752f36-console-config\" (UniqueName: \"kubernetes.io/configmap/8e7e744e-9806-48ca-9d02-94bb07752f36-console-config\") pod \"8e7e744e-9806-48ca-9d02-94bb07752f36\" (UID: \"8e7e744e-9806-48ca-9d02-94bb07752f36\") " Oct 30 01:18:23.877268 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:23.877237 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e7e744e-9806-48ca-9d02-94bb07752f36-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8e7e744e-9806-48ca-9d02-94bb07752f36-console-oauth-config\") pod \"8e7e744e-9806-48ca-9d02-94bb07752f36\" (UID: \"8e7e744e-9806-48ca-9d02-94bb07752f36\") " Oct 30 01:18:23.877439 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:23.877214 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7e744e-9806-48ca-9d02-94bb07752f36-tmp" pod "8e7e744e-9806-48ca-9d02-94bb07752f36" (UID: "8e7e744e-9806-48ca-9d02-94bb07752f36"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:18:23.877439 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:23.877372 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e7e744e-9806-48ca-9d02-94bb07752f36-service-ca" pod "8e7e744e-9806-48ca-9d02-94bb07752f36" (UID: "8e7e744e-9806-48ca-9d02-94bb07752f36"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:18:23.877655 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:23.877500 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e7e744e-9806-48ca-9d02-94bb07752f36-oauth-serving-cert" pod "8e7e744e-9806-48ca-9d02-94bb07752f36" (UID: "8e7e744e-9806-48ca-9d02-94bb07752f36"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:18:23.877655 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:23.877582 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e7e744e-9806-48ca-9d02-94bb07752f36-console-config" pod "8e7e744e-9806-48ca-9d02-94bb07752f36" (UID: "8e7e744e-9806-48ca-9d02-94bb07752f36"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:18:23.879151 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:23.879127 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e7e744e-9806-48ca-9d02-94bb07752f36-console-serving-cert" pod "8e7e744e-9806-48ca-9d02-94bb07752f36" (UID: "8e7e744e-9806-48ca-9d02-94bb07752f36"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:18:23.879314 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:23.879298 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e7e744e-9806-48ca-9d02-94bb07752f36-console-oauth-config" pod "8e7e744e-9806-48ca-9d02-94bb07752f36" (UID: "8e7e744e-9806-48ca-9d02-94bb07752f36"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:18:23.879367 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:23.879304 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e7e744e-9806-48ca-9d02-94bb07752f36-kube-api-access-zz5xc" pod "8e7e744e-9806-48ca-9d02-94bb07752f36" (UID: "8e7e744e-9806-48ca-9d02-94bb07752f36"). InnerVolumeSpecName "kube-api-access-zz5xc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:18:23.977850 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:23.977812 6221 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8e7e744e-9806-48ca-9d02-94bb07752f36-console-serving-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:18:23.977850 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:23.977843 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz5xc\" (UniqueName: \"kubernetes.io/projected/8e7e744e-9806-48ca-9d02-94bb07752f36-kube-api-access-zz5xc\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:18:23.978048 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:23.977858 6221 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8e7e744e-9806-48ca-9d02-94bb07752f36-console-config\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:18:23.978048 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:23.977871 6221 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8e7e744e-9806-48ca-9d02-94bb07752f36-console-oauth-config\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:18:23.978048 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:23.977882 6221 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8e7e744e-9806-48ca-9d02-94bb07752f36-service-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:18:23.978048 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:23.977894 6221 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8e7e744e-9806-48ca-9d02-94bb07752f36-tmp\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:18:23.978048 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:23.977903 6221 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8e7e744e-9806-48ca-9d02-94bb07752f36-oauth-serving-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:18:24.543727 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:24.543690 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db13cab2-9b76-44b5-8577-9f8f0d3f292e" path="/var/lib/kubelet/pods/db13cab2-9b76-44b5-8577-9f8f0d3f292e/volumes" Oct 30 01:18:24.693454 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:24.693419 6221 generic.go:356] "Generic (PLEG): container finished" podID="8e7e744e-9806-48ca-9d02-94bb07752f36" containerID="fc1bd0a7e19c1ff001def32af6a08009019658216ffa26169a4abeb435cdfeb0" exitCode=0 Oct 30 01:18:24.693950 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:24.693464 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9bf55565f-w7bkn" Oct 30 01:18:24.693950 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:24.693500 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9bf55565f-w7bkn" event={"ID":"8e7e744e-9806-48ca-9d02-94bb07752f36","Type":"ContainerDied","Data":"fc1bd0a7e19c1ff001def32af6a08009019658216ffa26169a4abeb435cdfeb0"} Oct 30 01:18:24.693950 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:24.693564 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9bf55565f-w7bkn" event={"ID":"8e7e744e-9806-48ca-9d02-94bb07752f36","Type":"ContainerDied","Data":"041c582efbcd1445b58cc13ef4091c174b65eb6004559fe5429f4cacc4235d4f"} Oct 30 01:18:24.693950 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:24.693588 6221 scope.go:122] "RemoveContainer" containerID="fc1bd0a7e19c1ff001def32af6a08009019658216ffa26169a4abeb435cdfeb0" Oct 30 01:18:24.702655 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:24.702611 6221 scope.go:122] "RemoveContainer" containerID="fc1bd0a7e19c1ff001def32af6a08009019658216ffa26169a4abeb435cdfeb0" Oct 30 01:18:24.703104 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:18:24.703079 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc1bd0a7e19c1ff001def32af6a08009019658216ffa26169a4abeb435cdfeb0\": container with ID starting with fc1bd0a7e19c1ff001def32af6a08009019658216ffa26169a4abeb435cdfeb0 not found: ID does not exist" containerID="fc1bd0a7e19c1ff001def32af6a08009019658216ffa26169a4abeb435cdfeb0" Oct 30 01:18:24.703179 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:24.703108 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc1bd0a7e19c1ff001def32af6a08009019658216ffa26169a4abeb435cdfeb0"} err="failed to get container status \"fc1bd0a7e19c1ff001def32af6a08009019658216ffa26169a4abeb435cdfeb0\": rpc error: code = NotFound desc = could not find container \"fc1bd0a7e19c1ff001def32af6a08009019658216ffa26169a4abeb435cdfeb0\": container with ID starting with fc1bd0a7e19c1ff001def32af6a08009019658216ffa26169a4abeb435cdfeb0 not found: ID does not exist" Oct 30 01:18:24.743181 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:24.743157 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9bf55565f-w7bkn"] Oct 30 01:18:24.755462 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:24.755439 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-9bf55565f-w7bkn"] Oct 30 01:18:26.544112 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:26.544077 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e7e744e-9806-48ca-9d02-94bb07752f36" path="/var/lib/kubelet/pods/8e7e744e-9806-48ca-9d02-94bb07752f36/volumes" Oct 30 01:18:33.661462 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:18:33.661428 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bbhx5" Oct 30 01:18:51.491163 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:18:51.491126 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:19:05.616727 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:05.616689 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5494f674b9-8wxhh"] Oct 30 01:19:05.617233 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:05.617041 6221 cpu_manager.go:414] "containerMap: removing container" podUID="db13cab2-9b76-44b5-8577-9f8f0d3f292e" containerName="console" Oct 30 01:19:05.617233 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:05.617060 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db13cab2-9b76-44b5-8577-9f8f0d3f292e" containerName="console" Oct 30 01:19:05.617233 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:05.617100 6221 cpu_manager.go:414] "containerMap: removing container" podUID="8e7e744e-9806-48ca-9d02-94bb07752f36" containerName="console" Oct 30 01:19:05.617233 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:05.617108 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e7e744e-9806-48ca-9d02-94bb07752f36" containerName="console" Oct 30 01:19:05.617233 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:05.617176 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e7e744e-9806-48ca-9d02-94bb07752f36" containerName="console" Oct 30 01:19:05.617233 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:05.617189 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="db13cab2-9b76-44b5-8577-9f8f0d3f292e" containerName="console" Oct 30 01:19:05.619941 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:05.619920 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5494f674b9-8wxhh" Oct 30 01:19:05.650382 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:05.650356 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5494f674b9-8wxhh"] Oct 30 01:19:05.661055 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:05.661031 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a2b1d96-2a19-4db8-9c44-674d62f0f0a4-trusted-ca-bundle\") pod \"console-5494f674b9-8wxhh\" (UID: \"9a2b1d96-2a19-4db8-9c44-674d62f0f0a4\") " pod="openshift-console/console-5494f674b9-8wxhh" Oct 30 01:19:05.661173 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:05.661066 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a2b1d96-2a19-4db8-9c44-674d62f0f0a4-service-ca\") pod \"console-5494f674b9-8wxhh\" (UID: \"9a2b1d96-2a19-4db8-9c44-674d62f0f0a4\") " pod="openshift-console/console-5494f674b9-8wxhh" Oct 30 01:19:05.661173 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:05.661097 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a2b1d96-2a19-4db8-9c44-674d62f0f0a4-console-serving-cert\") pod \"console-5494f674b9-8wxhh\" (UID: \"9a2b1d96-2a19-4db8-9c44-674d62f0f0a4\") " pod="openshift-console/console-5494f674b9-8wxhh" Oct 30 01:19:05.661249 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:05.661220 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a2b1d96-2a19-4db8-9c44-674d62f0f0a4-console-config\") pod \"console-5494f674b9-8wxhh\" (UID: \"9a2b1d96-2a19-4db8-9c44-674d62f0f0a4\") " pod="openshift-console/console-5494f674b9-8wxhh" Oct 30 01:19:05.661282 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:05.661268 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a2b1d96-2a19-4db8-9c44-674d62f0f0a4-oauth-serving-cert\") pod \"console-5494f674b9-8wxhh\" (UID: \"9a2b1d96-2a19-4db8-9c44-674d62f0f0a4\") " pod="openshift-console/console-5494f674b9-8wxhh" Oct 30 01:19:05.661342 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:05.661320 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z26mj\" (UniqueName: \"kubernetes.io/projected/9a2b1d96-2a19-4db8-9c44-674d62f0f0a4-kube-api-access-z26mj\") pod \"console-5494f674b9-8wxhh\" (UID: \"9a2b1d96-2a19-4db8-9c44-674d62f0f0a4\") " pod="openshift-console/console-5494f674b9-8wxhh" Oct 30 01:19:05.661385 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:05.661367 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a2b1d96-2a19-4db8-9c44-674d62f0f0a4-tmp\") pod \"console-5494f674b9-8wxhh\" (UID: \"9a2b1d96-2a19-4db8-9c44-674d62f0f0a4\") " pod="openshift-console/console-5494f674b9-8wxhh" Oct 30 01:19:05.661469 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:05.661439 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a2b1d96-2a19-4db8-9c44-674d62f0f0a4-console-oauth-config\") pod \"console-5494f674b9-8wxhh\" (UID: \"9a2b1d96-2a19-4db8-9c44-674d62f0f0a4\") " pod="openshift-console/console-5494f674b9-8wxhh" Oct 30 01:19:05.761835 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:05.761795 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a2b1d96-2a19-4db8-9c44-674d62f0f0a4-trusted-ca-bundle\") pod \"console-5494f674b9-8wxhh\" (UID: \"9a2b1d96-2a19-4db8-9c44-674d62f0f0a4\") " pod="openshift-console/console-5494f674b9-8wxhh" Oct 30 01:19:05.762037 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:05.761846 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a2b1d96-2a19-4db8-9c44-674d62f0f0a4-service-ca\") pod \"console-5494f674b9-8wxhh\" (UID: \"9a2b1d96-2a19-4db8-9c44-674d62f0f0a4\") " pod="openshift-console/console-5494f674b9-8wxhh" Oct 30 01:19:05.762037 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:05.761873 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a2b1d96-2a19-4db8-9c44-674d62f0f0a4-console-serving-cert\") pod \"console-5494f674b9-8wxhh\" (UID: \"9a2b1d96-2a19-4db8-9c44-674d62f0f0a4\") " pod="openshift-console/console-5494f674b9-8wxhh" Oct 30 01:19:05.762037 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:05.761891 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a2b1d96-2a19-4db8-9c44-674d62f0f0a4-console-config\") pod \"console-5494f674b9-8wxhh\" (UID: \"9a2b1d96-2a19-4db8-9c44-674d62f0f0a4\") " pod="openshift-console/console-5494f674b9-8wxhh" Oct 30 01:19:05.762037 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:05.761923 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a2b1d96-2a19-4db8-9c44-674d62f0f0a4-oauth-serving-cert\") pod \"console-5494f674b9-8wxhh\" (UID: \"9a2b1d96-2a19-4db8-9c44-674d62f0f0a4\") " pod="openshift-console/console-5494f674b9-8wxhh" Oct 30 01:19:05.762037 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:05.761940 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z26mj\" (UniqueName: \"kubernetes.io/projected/9a2b1d96-2a19-4db8-9c44-674d62f0f0a4-kube-api-access-z26mj\") pod \"console-5494f674b9-8wxhh\" (UID: \"9a2b1d96-2a19-4db8-9c44-674d62f0f0a4\") " pod="openshift-console/console-5494f674b9-8wxhh" Oct 30 01:19:05.762338 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:05.762307 6221 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 01:19:05.762396 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:05.762339 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a2b1d96-2a19-4db8-9c44-674d62f0f0a4-tmp\") pod \"console-5494f674b9-8wxhh\" (UID: \"9a2b1d96-2a19-4db8-9c44-674d62f0f0a4\") " pod="openshift-console/console-5494f674b9-8wxhh" Oct 30 01:19:05.762466 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:05.762405 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a2b1d96-2a19-4db8-9c44-674d62f0f0a4-console-oauth-config\") pod \"console-5494f674b9-8wxhh\" (UID: \"9a2b1d96-2a19-4db8-9c44-674d62f0f0a4\") " pod="openshift-console/console-5494f674b9-8wxhh" Oct 30 01:19:05.762775 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:05.762748 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a2b1d96-2a19-4db8-9c44-674d62f0f0a4-service-ca\") pod \"console-5494f674b9-8wxhh\" (UID: \"9a2b1d96-2a19-4db8-9c44-674d62f0f0a4\") " pod="openshift-console/console-5494f674b9-8wxhh" Oct 30 01:19:05.762916 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:05.762780 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a2b1d96-2a19-4db8-9c44-674d62f0f0a4-tmp\") pod \"console-5494f674b9-8wxhh\" (UID: \"9a2b1d96-2a19-4db8-9c44-674d62f0f0a4\") " pod="openshift-console/console-5494f674b9-8wxhh" Oct 30 01:19:05.762916 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:05.762774 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a2b1d96-2a19-4db8-9c44-674d62f0f0a4-console-config\") pod \"console-5494f674b9-8wxhh\" (UID: \"9a2b1d96-2a19-4db8-9c44-674d62f0f0a4\") " pod="openshift-console/console-5494f674b9-8wxhh" Oct 30 01:19:05.763028 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:05.762958 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a2b1d96-2a19-4db8-9c44-674d62f0f0a4-trusted-ca-bundle\") pod \"console-5494f674b9-8wxhh\" (UID: \"9a2b1d96-2a19-4db8-9c44-674d62f0f0a4\") " pod="openshift-console/console-5494f674b9-8wxhh" Oct 30 01:19:05.763028 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:05.762992 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a2b1d96-2a19-4db8-9c44-674d62f0f0a4-oauth-serving-cert\") pod \"console-5494f674b9-8wxhh\" (UID: \"9a2b1d96-2a19-4db8-9c44-674d62f0f0a4\") " pod="openshift-console/console-5494f674b9-8wxhh" Oct 30 01:19:05.764844 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:05.764825 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a2b1d96-2a19-4db8-9c44-674d62f0f0a4-console-oauth-config\") pod \"console-5494f674b9-8wxhh\" (UID: \"9a2b1d96-2a19-4db8-9c44-674d62f0f0a4\") " pod="openshift-console/console-5494f674b9-8wxhh" Oct 30 01:19:05.764960 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:05.764944 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a2b1d96-2a19-4db8-9c44-674d62f0f0a4-console-serving-cert\") pod \"console-5494f674b9-8wxhh\" (UID: \"9a2b1d96-2a19-4db8-9c44-674d62f0f0a4\") " pod="openshift-console/console-5494f674b9-8wxhh" Oct 30 01:19:05.780674 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:05.780653 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z26mj\" (UniqueName: \"kubernetes.io/projected/9a2b1d96-2a19-4db8-9c44-674d62f0f0a4-kube-api-access-z26mj\") pod \"console-5494f674b9-8wxhh\" (UID: \"9a2b1d96-2a19-4db8-9c44-674d62f0f0a4\") " pod="openshift-console/console-5494f674b9-8wxhh" Oct 30 01:19:05.930029 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:05.929951 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5494f674b9-8wxhh" Oct 30 01:19:06.236929 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:06.236855 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5494f674b9-8wxhh"] Oct 30 01:19:06.242875 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:19:06.242848 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a2b1d96_2a19_4db8_9c44_674d62f0f0a4.slice/crio-903f47a9c8e34c18c61e65c42617c4b21bccac087255eeb06875dfe714910b5a WatchSource:0}: Error finding container 903f47a9c8e34c18c61e65c42617c4b21bccac087255eeb06875dfe714910b5a: Status 404 returned error can't find the container with id 903f47a9c8e34c18c61e65c42617c4b21bccac087255eeb06875dfe714910b5a Oct 30 01:19:06.833006 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:06.832949 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5494f674b9-8wxhh" event={"ID":"9a2b1d96-2a19-4db8-9c44-674d62f0f0a4","Type":"ContainerStarted","Data":"c3052f9c9054bc5d72100d3d9047d7327e3624a852faad7f5538e91f8c6b5a48"} Oct 30 01:19:06.833507 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:06.833022 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5494f674b9-8wxhh" event={"ID":"9a2b1d96-2a19-4db8-9c44-674d62f0f0a4","Type":"ContainerStarted","Data":"903f47a9c8e34c18c61e65c42617c4b21bccac087255eeb06875dfe714910b5a"} Oct 30 01:19:06.885806 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:06.885756 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5494f674b9-8wxhh" podStartSLOduration=1.885744788 podStartE2EDuration="1.885744788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:06.884325228 +0000 UTC m=+70.462612564" watchObservedRunningTime="2026-10-30 01:19:06.885744788 +0000 UTC m=+70.464032121" Oct 30 01:19:09.259074 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:09.259039 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 01:19:09.259839 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:09.259403 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d205185b-fc45-4a76-9719-20b00107abf1" containerName="alertmanager" containerID="cri-o://7c4f99037568d2ad4c8bac57b44357ea2bef68911bc930f4dd8ff2301cb02b04" gracePeriod=120 Oct 30 01:19:09.259839 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:09.259464 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d205185b-fc45-4a76-9719-20b00107abf1" containerName="kube-rbac-proxy-metric" containerID="cri-o://faecbc4dc56aabdfd4bddaf0013c0303a9c2ca18ac5b800ae1bdcff268fa6d94" gracePeriod=120 Oct 30 01:19:09.259839 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:09.259481 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d205185b-fc45-4a76-9719-20b00107abf1" containerName="kube-rbac-proxy-web" containerID="cri-o://5e56bb5fc49748e897907a5d67bde94bb556b6c08dcd80e36eb70b7bcb98ee74" gracePeriod=120 Oct 30 01:19:09.259839 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:09.259500 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d205185b-fc45-4a76-9719-20b00107abf1" containerName="config-reloader" containerID="cri-o://087568965eb285b660987b3f2418bff23f91bd55c25f5c9b6abc10903538f624" gracePeriod=120 Oct 30 01:19:09.259839 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:09.259544 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d205185b-fc45-4a76-9719-20b00107abf1" containerName="kube-rbac-proxy" containerID="cri-o://2f312c75cad0960abe3ae70749c0cbae1dc1a744e61c9619efc01a84f6cca31b" gracePeriod=120 Oct 30 01:19:09.259839 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:09.259550 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d205185b-fc45-4a76-9719-20b00107abf1" containerName="prom-label-proxy" containerID="cri-o://b7e6935898f0ee20cc221ace812df50f79bc1400474593d50e2ce45b44280d47" gracePeriod=120 Oct 30 01:19:09.846824 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:09.846791 6221 generic.go:356] "Generic (PLEG): container finished" podID="d205185b-fc45-4a76-9719-20b00107abf1" containerID="b7e6935898f0ee20cc221ace812df50f79bc1400474593d50e2ce45b44280d47" exitCode=0 Oct 30 01:19:09.846824 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:09.846817 6221 generic.go:356] "Generic (PLEG): container finished" podID="d205185b-fc45-4a76-9719-20b00107abf1" containerID="2f312c75cad0960abe3ae70749c0cbae1dc1a744e61c9619efc01a84f6cca31b" exitCode=0 Oct 30 01:19:09.846824 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:09.846825 6221 generic.go:356] "Generic (PLEG): container finished" podID="d205185b-fc45-4a76-9719-20b00107abf1" containerID="087568965eb285b660987b3f2418bff23f91bd55c25f5c9b6abc10903538f624" exitCode=0 Oct 30 01:19:09.846824 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:09.846832 6221 generic.go:356] "Generic (PLEG): container finished" podID="d205185b-fc45-4a76-9719-20b00107abf1" containerID="7c4f99037568d2ad4c8bac57b44357ea2bef68911bc930f4dd8ff2301cb02b04" exitCode=0 Oct 30 01:19:09.847173 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:09.846867 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d205185b-fc45-4a76-9719-20b00107abf1","Type":"ContainerDied","Data":"b7e6935898f0ee20cc221ace812df50f79bc1400474593d50e2ce45b44280d47"} Oct 30 01:19:09.847173 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:09.846930 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d205185b-fc45-4a76-9719-20b00107abf1","Type":"ContainerDied","Data":"2f312c75cad0960abe3ae70749c0cbae1dc1a744e61c9619efc01a84f6cca31b"} Oct 30 01:19:09.847173 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:09.846956 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d205185b-fc45-4a76-9719-20b00107abf1","Type":"ContainerDied","Data":"087568965eb285b660987b3f2418bff23f91bd55c25f5c9b6abc10903538f624"} Oct 30 01:19:09.847173 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:09.846982 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d205185b-fc45-4a76-9719-20b00107abf1","Type":"ContainerDied","Data":"7c4f99037568d2ad4c8bac57b44357ea2bef68911bc930f4dd8ff2301cb02b04"} Oct 30 01:19:10.511524 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.511496 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:19:10.597247 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.597059 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d205185b-fc45-4a76-9719-20b00107abf1-tls-assets\" (UniqueName: \"kubernetes.io/projected/d205185b-fc45-4a76-9719-20b00107abf1-tls-assets\") pod \"d205185b-fc45-4a76-9719-20b00107abf1\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " Oct 30 01:19:10.597247 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.597151 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d205185b-fc45-4a76-9719-20b00107abf1-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d205185b-fc45-4a76-9719-20b00107abf1-metrics-client-ca\") pod \"d205185b-fc45-4a76-9719-20b00107abf1\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " Oct 30 01:19:10.597247 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.597232 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d205185b-fc45-4a76-9719-20b00107abf1-kube-api-access-mcd88\" (UniqueName: \"kubernetes.io/projected/d205185b-fc45-4a76-9719-20b00107abf1-kube-api-access-mcd88\") pod \"d205185b-fc45-4a76-9719-20b00107abf1\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " Oct 30 01:19:10.598428 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.597267 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-secret-alertmanager-kube-rbac-proxy-web\") pod \"d205185b-fc45-4a76-9719-20b00107abf1\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " Oct 30 01:19:10.598428 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.597297 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d205185b-fc45-4a76-9719-20b00107abf1-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d205185b-fc45-4a76-9719-20b00107abf1-alertmanager-main-db\") pod \"d205185b-fc45-4a76-9719-20b00107abf1\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " Oct 30 01:19:10.598428 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.597475 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d205185b-fc45-4a76-9719-20b00107abf1-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d205185b-fc45-4a76-9719-20b00107abf1-alertmanager-trusted-ca-bundle\") pod \"d205185b-fc45-4a76-9719-20b00107abf1\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " Oct 30 01:19:10.598428 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.597639 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"d205185b-fc45-4a76-9719-20b00107abf1\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " Oct 30 01:19:10.598428 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.597787 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-secret-alertmanager-kube-rbac-proxy\") pod \"d205185b-fc45-4a76-9719-20b00107abf1\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " Oct 30 01:19:10.598428 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.597820 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-config-volume\" (UniqueName: \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-config-volume\") pod \"d205185b-fc45-4a76-9719-20b00107abf1\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " Oct 30 01:19:10.598428 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.597957 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-web-config\" (UniqueName: \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-web-config\") pod \"d205185b-fc45-4a76-9719-20b00107abf1\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " Oct 30 01:19:10.598428 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.598378 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-secret-alertmanager-main-tls\") pod \"d205185b-fc45-4a76-9719-20b00107abf1\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " Oct 30 01:19:10.598428 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.598420 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d205185b-fc45-4a76-9719-20b00107abf1-config-out\" (UniqueName: \"kubernetes.io/empty-dir/d205185b-fc45-4a76-9719-20b00107abf1-config-out\") pod \"d205185b-fc45-4a76-9719-20b00107abf1\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " Oct 30 01:19:10.598887 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.598448 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-cluster-tls-config\") pod \"d205185b-fc45-4a76-9719-20b00107abf1\" (UID: \"d205185b-fc45-4a76-9719-20b00107abf1\") " Oct 30 01:19:10.599443 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.599412 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d205185b-fc45-4a76-9719-20b00107abf1-metrics-client-ca" pod "d205185b-fc45-4a76-9719-20b00107abf1" (UID: "d205185b-fc45-4a76-9719-20b00107abf1"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:10.600321 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.600286 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d205185b-fc45-4a76-9719-20b00107abf1-alertmanager-main-db" pod "d205185b-fc45-4a76-9719-20b00107abf1" (UID: "d205185b-fc45-4a76-9719-20b00107abf1"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:10.600490 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.600466 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d205185b-fc45-4a76-9719-20b00107abf1-alertmanager-trusted-ca-bundle" pod "d205185b-fc45-4a76-9719-20b00107abf1" (UID: "d205185b-fc45-4a76-9719-20b00107abf1"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:10.601187 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.601125 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-secret-alertmanager-kube-rbac-proxy" pod "d205185b-fc45-4a76-9719-20b00107abf1" (UID: "d205185b-fc45-4a76-9719-20b00107abf1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:19:10.601686 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.601608 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d205185b-fc45-4a76-9719-20b00107abf1-tls-assets" pod "d205185b-fc45-4a76-9719-20b00107abf1" (UID: "d205185b-fc45-4a76-9719-20b00107abf1"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:10.603338 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.603312 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d205185b-fc45-4a76-9719-20b00107abf1-config-out" pod "d205185b-fc45-4a76-9719-20b00107abf1" (UID: "d205185b-fc45-4a76-9719-20b00107abf1"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:10.604716 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.604686 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-secret-alertmanager-kube-rbac-proxy-web" pod "d205185b-fc45-4a76-9719-20b00107abf1" (UID: "d205185b-fc45-4a76-9719-20b00107abf1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:19:10.606101 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.606035 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-secret-alertmanager-kube-rbac-proxy-metric" pod "d205185b-fc45-4a76-9719-20b00107abf1" (UID: "d205185b-fc45-4a76-9719-20b00107abf1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:19:10.606912 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.606875 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-config-volume" pod "d205185b-fc45-4a76-9719-20b00107abf1" (UID: "d205185b-fc45-4a76-9719-20b00107abf1"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:19:10.607000 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.606939 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d205185b-fc45-4a76-9719-20b00107abf1-kube-api-access-mcd88" pod "d205185b-fc45-4a76-9719-20b00107abf1" (UID: "d205185b-fc45-4a76-9719-20b00107abf1"). InnerVolumeSpecName "kube-api-access-mcd88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:10.608667 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.608635 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-secret-alertmanager-main-tls" pod "d205185b-fc45-4a76-9719-20b00107abf1" (UID: "d205185b-fc45-4a76-9719-20b00107abf1"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:19:10.608835 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.608814 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-cluster-tls-config" pod "d205185b-fc45-4a76-9719-20b00107abf1" (UID: "d205185b-fc45-4a76-9719-20b00107abf1"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:19:10.613336 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.613312 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-web-config" pod "d205185b-fc45-4a76-9719-20b00107abf1" (UID: "d205185b-fc45-4a76-9719-20b00107abf1"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:19:10.699966 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.699929 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcd88\" (UniqueName: \"kubernetes.io/projected/d205185b-fc45-4a76-9719-20b00107abf1-kube-api-access-mcd88\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:19:10.699966 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.699960 6221 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:19:10.699966 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.699971 6221 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d205185b-fc45-4a76-9719-20b00107abf1-alertmanager-main-db\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:19:10.700191 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.699979 6221 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d205185b-fc45-4a76-9719-20b00107abf1-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:19:10.700191 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.699991 6221 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:19:10.700191 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.700000 6221 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:19:10.700191 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.700012 6221 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:19:10.700191 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.700020 6221 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-web-config\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:19:10.700191 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.700028 6221 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-secret-alertmanager-main-tls\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:19:10.700191 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.700036 6221 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d205185b-fc45-4a76-9719-20b00107abf1-config-out\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:19:10.700191 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.700044 6221 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d205185b-fc45-4a76-9719-20b00107abf1-cluster-tls-config\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:19:10.700191 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.700053 6221 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d205185b-fc45-4a76-9719-20b00107abf1-tls-assets\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:19:10.700191 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.700061 6221 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d205185b-fc45-4a76-9719-20b00107abf1-metrics-client-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:19:10.852789 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.852711 6221 generic.go:356] "Generic (PLEG): container finished" podID="d205185b-fc45-4a76-9719-20b00107abf1" containerID="faecbc4dc56aabdfd4bddaf0013c0303a9c2ca18ac5b800ae1bdcff268fa6d94" exitCode=0 Oct 30 01:19:10.852789 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.852735 6221 generic.go:356] "Generic (PLEG): container finished" podID="d205185b-fc45-4a76-9719-20b00107abf1" containerID="5e56bb5fc49748e897907a5d67bde94bb556b6c08dcd80e36eb70b7bcb98ee74" exitCode=0 Oct 30 01:19:10.852957 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.852794 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:19:10.852957 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.852793 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d205185b-fc45-4a76-9719-20b00107abf1","Type":"ContainerDied","Data":"faecbc4dc56aabdfd4bddaf0013c0303a9c2ca18ac5b800ae1bdcff268fa6d94"} Oct 30 01:19:10.852957 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.852877 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d205185b-fc45-4a76-9719-20b00107abf1","Type":"ContainerDied","Data":"5e56bb5fc49748e897907a5d67bde94bb556b6c08dcd80e36eb70b7bcb98ee74"} Oct 30 01:19:10.852957 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.852921 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d205185b-fc45-4a76-9719-20b00107abf1","Type":"ContainerDied","Data":"66cfed6041737b855c39214ebc841b78bd2197bc4a3cbd61dfec11a91da2bde2"} Oct 30 01:19:10.852957 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.852946 6221 scope.go:122] "RemoveContainer" containerID="b7e6935898f0ee20cc221ace812df50f79bc1400474593d50e2ce45b44280d47" Oct 30 01:19:10.860055 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.860037 6221 scope.go:122] "RemoveContainer" containerID="faecbc4dc56aabdfd4bddaf0013c0303a9c2ca18ac5b800ae1bdcff268fa6d94" Oct 30 01:19:10.866570 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.866554 6221 scope.go:122] "RemoveContainer" containerID="2f312c75cad0960abe3ae70749c0cbae1dc1a744e61c9619efc01a84f6cca31b" Oct 30 01:19:10.872936 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.872921 6221 scope.go:122] "RemoveContainer" containerID="5e56bb5fc49748e897907a5d67bde94bb556b6c08dcd80e36eb70b7bcb98ee74" Oct 30 01:19:10.878935 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.878918 6221 scope.go:122] "RemoveContainer" containerID="087568965eb285b660987b3f2418bff23f91bd55c25f5c9b6abc10903538f624" Oct 30 01:19:10.885599 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.885574 6221 scope.go:122] "RemoveContainer" containerID="7c4f99037568d2ad4c8bac57b44357ea2bef68911bc930f4dd8ff2301cb02b04" Oct 30 01:19:10.891856 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.891831 6221 scope.go:122] "RemoveContainer" containerID="fa086018fcdc7e625873376ddc1edd48c8c59518991ac6ffee80dc216b9ef4af" Oct 30 01:19:10.898015 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.897996 6221 scope.go:122] "RemoveContainer" containerID="b7e6935898f0ee20cc221ace812df50f79bc1400474593d50e2ce45b44280d47" Oct 30 01:19:10.898292 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:19:10.898272 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7e6935898f0ee20cc221ace812df50f79bc1400474593d50e2ce45b44280d47\": container with ID starting with b7e6935898f0ee20cc221ace812df50f79bc1400474593d50e2ce45b44280d47 not found: ID does not exist" containerID="b7e6935898f0ee20cc221ace812df50f79bc1400474593d50e2ce45b44280d47" Oct 30 01:19:10.898368 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.898301 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7e6935898f0ee20cc221ace812df50f79bc1400474593d50e2ce45b44280d47"} err="failed to get container status \"b7e6935898f0ee20cc221ace812df50f79bc1400474593d50e2ce45b44280d47\": rpc error: code = NotFound desc = could not find container \"b7e6935898f0ee20cc221ace812df50f79bc1400474593d50e2ce45b44280d47\": container with ID starting with b7e6935898f0ee20cc221ace812df50f79bc1400474593d50e2ce45b44280d47 not found: ID does not exist" Oct 30 01:19:10.898368 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.898318 6221 scope.go:122] "RemoveContainer" containerID="faecbc4dc56aabdfd4bddaf0013c0303a9c2ca18ac5b800ae1bdcff268fa6d94" Oct 30 01:19:10.898569 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:19:10.898550 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faecbc4dc56aabdfd4bddaf0013c0303a9c2ca18ac5b800ae1bdcff268fa6d94\": container with ID starting with faecbc4dc56aabdfd4bddaf0013c0303a9c2ca18ac5b800ae1bdcff268fa6d94 not found: ID does not exist" containerID="faecbc4dc56aabdfd4bddaf0013c0303a9c2ca18ac5b800ae1bdcff268fa6d94" Oct 30 01:19:10.898639 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.898579 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faecbc4dc56aabdfd4bddaf0013c0303a9c2ca18ac5b800ae1bdcff268fa6d94"} err="failed to get container status \"faecbc4dc56aabdfd4bddaf0013c0303a9c2ca18ac5b800ae1bdcff268fa6d94\": rpc error: code = NotFound desc = could not find container \"faecbc4dc56aabdfd4bddaf0013c0303a9c2ca18ac5b800ae1bdcff268fa6d94\": container with ID starting with faecbc4dc56aabdfd4bddaf0013c0303a9c2ca18ac5b800ae1bdcff268fa6d94 not found: ID does not exist" Oct 30 01:19:10.898639 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.898601 6221 scope.go:122] "RemoveContainer" containerID="2f312c75cad0960abe3ae70749c0cbae1dc1a744e61c9619efc01a84f6cca31b" Oct 30 01:19:10.898837 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:19:10.898821 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f312c75cad0960abe3ae70749c0cbae1dc1a744e61c9619efc01a84f6cca31b\": container with ID starting with 2f312c75cad0960abe3ae70749c0cbae1dc1a744e61c9619efc01a84f6cca31b not found: ID does not exist" containerID="2f312c75cad0960abe3ae70749c0cbae1dc1a744e61c9619efc01a84f6cca31b" Oct 30 01:19:10.898898 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.898846 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f312c75cad0960abe3ae70749c0cbae1dc1a744e61c9619efc01a84f6cca31b"} err="failed to get container status \"2f312c75cad0960abe3ae70749c0cbae1dc1a744e61c9619efc01a84f6cca31b\": rpc error: code = NotFound desc = could not find container \"2f312c75cad0960abe3ae70749c0cbae1dc1a744e61c9619efc01a84f6cca31b\": container with ID starting with 2f312c75cad0960abe3ae70749c0cbae1dc1a744e61c9619efc01a84f6cca31b not found: ID does not exist" Oct 30 01:19:10.898898 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.898865 6221 scope.go:122] "RemoveContainer" containerID="5e56bb5fc49748e897907a5d67bde94bb556b6c08dcd80e36eb70b7bcb98ee74" Oct 30 01:19:10.899128 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:19:10.899111 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e56bb5fc49748e897907a5d67bde94bb556b6c08dcd80e36eb70b7bcb98ee74\": container with ID starting with 5e56bb5fc49748e897907a5d67bde94bb556b6c08dcd80e36eb70b7bcb98ee74 not found: ID does not exist" containerID="5e56bb5fc49748e897907a5d67bde94bb556b6c08dcd80e36eb70b7bcb98ee74" Oct 30 01:19:10.899165 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.899136 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e56bb5fc49748e897907a5d67bde94bb556b6c08dcd80e36eb70b7bcb98ee74"} err="failed to get container status \"5e56bb5fc49748e897907a5d67bde94bb556b6c08dcd80e36eb70b7bcb98ee74\": rpc error: code = NotFound desc = could not find container \"5e56bb5fc49748e897907a5d67bde94bb556b6c08dcd80e36eb70b7bcb98ee74\": container with ID starting with 5e56bb5fc49748e897907a5d67bde94bb556b6c08dcd80e36eb70b7bcb98ee74 not found: ID does not exist" Oct 30 01:19:10.899165 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.899150 6221 scope.go:122] "RemoveContainer" containerID="087568965eb285b660987b3f2418bff23f91bd55c25f5c9b6abc10903538f624" Oct 30 01:19:10.899357 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:19:10.899340 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"087568965eb285b660987b3f2418bff23f91bd55c25f5c9b6abc10903538f624\": container with ID starting with 087568965eb285b660987b3f2418bff23f91bd55c25f5c9b6abc10903538f624 not found: ID does not exist" containerID="087568965eb285b660987b3f2418bff23f91bd55c25f5c9b6abc10903538f624" Oct 30 01:19:10.899436 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.899363 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"087568965eb285b660987b3f2418bff23f91bd55c25f5c9b6abc10903538f624"} err="failed to get container status \"087568965eb285b660987b3f2418bff23f91bd55c25f5c9b6abc10903538f624\": rpc error: code = NotFound desc = could not find container \"087568965eb285b660987b3f2418bff23f91bd55c25f5c9b6abc10903538f624\": container with ID starting with 087568965eb285b660987b3f2418bff23f91bd55c25f5c9b6abc10903538f624 not found: ID does not exist" Oct 30 01:19:10.899436 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.899381 6221 scope.go:122] "RemoveContainer" containerID="7c4f99037568d2ad4c8bac57b44357ea2bef68911bc930f4dd8ff2301cb02b04" Oct 30 01:19:10.899597 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:19:10.899578 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c4f99037568d2ad4c8bac57b44357ea2bef68911bc930f4dd8ff2301cb02b04\": container with ID starting with 7c4f99037568d2ad4c8bac57b44357ea2bef68911bc930f4dd8ff2301cb02b04 not found: ID does not exist" containerID="7c4f99037568d2ad4c8bac57b44357ea2bef68911bc930f4dd8ff2301cb02b04" Oct 30 01:19:10.899640 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.899604 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c4f99037568d2ad4c8bac57b44357ea2bef68911bc930f4dd8ff2301cb02b04"} err="failed to get container status \"7c4f99037568d2ad4c8bac57b44357ea2bef68911bc930f4dd8ff2301cb02b04\": rpc error: code = NotFound desc = could not find container \"7c4f99037568d2ad4c8bac57b44357ea2bef68911bc930f4dd8ff2301cb02b04\": container with ID starting with 7c4f99037568d2ad4c8bac57b44357ea2bef68911bc930f4dd8ff2301cb02b04 not found: ID does not exist" Oct 30 01:19:10.899640 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.899623 6221 scope.go:122] "RemoveContainer" containerID="fa086018fcdc7e625873376ddc1edd48c8c59518991ac6ffee80dc216b9ef4af" Oct 30 01:19:10.899857 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:19:10.899841 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa086018fcdc7e625873376ddc1edd48c8c59518991ac6ffee80dc216b9ef4af\": container with ID starting with fa086018fcdc7e625873376ddc1edd48c8c59518991ac6ffee80dc216b9ef4af not found: ID does not exist" containerID="fa086018fcdc7e625873376ddc1edd48c8c59518991ac6ffee80dc216b9ef4af" Oct 30 01:19:10.899939 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.899863 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa086018fcdc7e625873376ddc1edd48c8c59518991ac6ffee80dc216b9ef4af"} err="failed to get container status \"fa086018fcdc7e625873376ddc1edd48c8c59518991ac6ffee80dc216b9ef4af\": rpc error: code = NotFound desc = could not find container \"fa086018fcdc7e625873376ddc1edd48c8c59518991ac6ffee80dc216b9ef4af\": container with ID starting with fa086018fcdc7e625873376ddc1edd48c8c59518991ac6ffee80dc216b9ef4af not found: ID does not exist" Oct 30 01:19:10.899939 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.899882 6221 scope.go:122] "RemoveContainer" containerID="b7e6935898f0ee20cc221ace812df50f79bc1400474593d50e2ce45b44280d47" Oct 30 01:19:10.900157 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.900140 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7e6935898f0ee20cc221ace812df50f79bc1400474593d50e2ce45b44280d47"} err="failed to get container status \"b7e6935898f0ee20cc221ace812df50f79bc1400474593d50e2ce45b44280d47\": rpc error: code = NotFound desc = could not find container \"b7e6935898f0ee20cc221ace812df50f79bc1400474593d50e2ce45b44280d47\": container with ID starting with b7e6935898f0ee20cc221ace812df50f79bc1400474593d50e2ce45b44280d47 not found: ID does not exist" Oct 30 01:19:10.900213 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.900157 6221 scope.go:122] "RemoveContainer" containerID="faecbc4dc56aabdfd4bddaf0013c0303a9c2ca18ac5b800ae1bdcff268fa6d94" Oct 30 01:19:10.900363 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.900345 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faecbc4dc56aabdfd4bddaf0013c0303a9c2ca18ac5b800ae1bdcff268fa6d94"} err="failed to get container status \"faecbc4dc56aabdfd4bddaf0013c0303a9c2ca18ac5b800ae1bdcff268fa6d94\": rpc error: code = NotFound desc = could not find container \"faecbc4dc56aabdfd4bddaf0013c0303a9c2ca18ac5b800ae1bdcff268fa6d94\": container with ID starting with faecbc4dc56aabdfd4bddaf0013c0303a9c2ca18ac5b800ae1bdcff268fa6d94 not found: ID does not exist" Oct 30 01:19:10.900410 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.900364 6221 scope.go:122] "RemoveContainer" containerID="2f312c75cad0960abe3ae70749c0cbae1dc1a744e61c9619efc01a84f6cca31b" Oct 30 01:19:10.900621 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.900593 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f312c75cad0960abe3ae70749c0cbae1dc1a744e61c9619efc01a84f6cca31b"} err="failed to get container status \"2f312c75cad0960abe3ae70749c0cbae1dc1a744e61c9619efc01a84f6cca31b\": rpc error: code = NotFound desc = could not find container \"2f312c75cad0960abe3ae70749c0cbae1dc1a744e61c9619efc01a84f6cca31b\": container with ID starting with 2f312c75cad0960abe3ae70749c0cbae1dc1a744e61c9619efc01a84f6cca31b not found: ID does not exist" Oct 30 01:19:10.900621 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.900613 6221 scope.go:122] "RemoveContainer" containerID="5e56bb5fc49748e897907a5d67bde94bb556b6c08dcd80e36eb70b7bcb98ee74" Oct 30 01:19:10.900833 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.900817 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e56bb5fc49748e897907a5d67bde94bb556b6c08dcd80e36eb70b7bcb98ee74"} err="failed to get container status \"5e56bb5fc49748e897907a5d67bde94bb556b6c08dcd80e36eb70b7bcb98ee74\": rpc error: code = NotFound desc = could not find container \"5e56bb5fc49748e897907a5d67bde94bb556b6c08dcd80e36eb70b7bcb98ee74\": container with ID starting with 5e56bb5fc49748e897907a5d67bde94bb556b6c08dcd80e36eb70b7bcb98ee74 not found: ID does not exist" Oct 30 01:19:10.900876 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.900833 6221 scope.go:122] "RemoveContainer" containerID="087568965eb285b660987b3f2418bff23f91bd55c25f5c9b6abc10903538f624" Oct 30 01:19:10.901034 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.901017 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"087568965eb285b660987b3f2418bff23f91bd55c25f5c9b6abc10903538f624"} err="failed to get container status \"087568965eb285b660987b3f2418bff23f91bd55c25f5c9b6abc10903538f624\": rpc error: code = NotFound desc = could not find container \"087568965eb285b660987b3f2418bff23f91bd55c25f5c9b6abc10903538f624\": container with ID starting with 087568965eb285b660987b3f2418bff23f91bd55c25f5c9b6abc10903538f624 not found: ID does not exist" Oct 30 01:19:10.901101 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.901035 6221 scope.go:122] "RemoveContainer" containerID="7c4f99037568d2ad4c8bac57b44357ea2bef68911bc930f4dd8ff2301cb02b04" Oct 30 01:19:10.901263 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.901244 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c4f99037568d2ad4c8bac57b44357ea2bef68911bc930f4dd8ff2301cb02b04"} err="failed to get container status \"7c4f99037568d2ad4c8bac57b44357ea2bef68911bc930f4dd8ff2301cb02b04\": rpc error: code = NotFound desc = could not find container \"7c4f99037568d2ad4c8bac57b44357ea2bef68911bc930f4dd8ff2301cb02b04\": container with ID starting with 7c4f99037568d2ad4c8bac57b44357ea2bef68911bc930f4dd8ff2301cb02b04 not found: ID does not exist" Oct 30 01:19:10.901321 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.901264 6221 scope.go:122] "RemoveContainer" containerID="fa086018fcdc7e625873376ddc1edd48c8c59518991ac6ffee80dc216b9ef4af" Oct 30 01:19:10.901469 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.901451 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa086018fcdc7e625873376ddc1edd48c8c59518991ac6ffee80dc216b9ef4af"} err="failed to get container status \"fa086018fcdc7e625873376ddc1edd48c8c59518991ac6ffee80dc216b9ef4af\": rpc error: code = NotFound desc = could not find container \"fa086018fcdc7e625873376ddc1edd48c8c59518991ac6ffee80dc216b9ef4af\": container with ID starting with fa086018fcdc7e625873376ddc1edd48c8c59518991ac6ffee80dc216b9ef4af not found: ID does not exist" Oct 30 01:19:10.905019 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.904997 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 01:19:10.917398 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.917378 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 01:19:10.937369 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.937347 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 01:19:10.937989 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.937968 6221 cpu_manager.go:414] "containerMap: removing container" podUID="d205185b-fc45-4a76-9719-20b00107abf1" containerName="prom-label-proxy" Oct 30 01:19:10.938160 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.938145 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d205185b-fc45-4a76-9719-20b00107abf1" containerName="prom-label-proxy" Oct 30 01:19:10.938263 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.938253 6221 cpu_manager.go:414] "containerMap: removing container" podUID="d205185b-fc45-4a76-9719-20b00107abf1" containerName="init-config-reloader" Oct 30 01:19:10.938337 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.938328 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d205185b-fc45-4a76-9719-20b00107abf1" containerName="init-config-reloader" Oct 30 01:19:10.938411 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.938402 6221 cpu_manager.go:414] "containerMap: removing container" podUID="d205185b-fc45-4a76-9719-20b00107abf1" containerName="kube-rbac-proxy-metric" Oct 30 01:19:10.938476 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.938466 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d205185b-fc45-4a76-9719-20b00107abf1" containerName="kube-rbac-proxy-metric" Oct 30 01:19:10.938569 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.938558 6221 cpu_manager.go:414] "containerMap: removing container" podUID="d205185b-fc45-4a76-9719-20b00107abf1" containerName="config-reloader" Oct 30 01:19:10.938656 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.938644 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d205185b-fc45-4a76-9719-20b00107abf1" containerName="config-reloader" Oct 30 01:19:10.938734 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.938726 6221 cpu_manager.go:414] "containerMap: removing container" podUID="d205185b-fc45-4a76-9719-20b00107abf1" containerName="kube-rbac-proxy" Oct 30 01:19:10.938798 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.938789 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d205185b-fc45-4a76-9719-20b00107abf1" containerName="kube-rbac-proxy" Oct 30 01:19:10.938860 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.938851 6221 cpu_manager.go:414] "containerMap: removing container" podUID="d205185b-fc45-4a76-9719-20b00107abf1" containerName="kube-rbac-proxy-web" Oct 30 01:19:10.938923 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.938914 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d205185b-fc45-4a76-9719-20b00107abf1" containerName="kube-rbac-proxy-web" Oct 30 01:19:10.939000 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.938992 6221 cpu_manager.go:414] "containerMap: removing container" podUID="d205185b-fc45-4a76-9719-20b00107abf1" containerName="alertmanager" Oct 30 01:19:10.939067 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.939058 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d205185b-fc45-4a76-9719-20b00107abf1" containerName="alertmanager" Oct 30 01:19:10.939253 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.939243 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="d205185b-fc45-4a76-9719-20b00107abf1" containerName="prom-label-proxy" Oct 30 01:19:10.939327 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.939319 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="d205185b-fc45-4a76-9719-20b00107abf1" containerName="init-config-reloader" Oct 30 01:19:10.939396 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.939388 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="d205185b-fc45-4a76-9719-20b00107abf1" containerName="kube-rbac-proxy-metric" Oct 30 01:19:10.939462 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.939454 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="d205185b-fc45-4a76-9719-20b00107abf1" containerName="config-reloader" Oct 30 01:19:10.939543 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.939519 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="d205185b-fc45-4a76-9719-20b00107abf1" containerName="kube-rbac-proxy" Oct 30 01:19:10.939618 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.939608 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="d205185b-fc45-4a76-9719-20b00107abf1" containerName="kube-rbac-proxy-web" Oct 30 01:19:10.939682 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.939674 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="d205185b-fc45-4a76-9719-20b00107abf1" containerName="alertmanager" Oct 30 01:19:10.943966 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.943949 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:19:10.961387 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.961231 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 01:19:10.961387 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.961256 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 01:19:10.962126 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.961906 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 01:19:10.962126 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.961947 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 01:19:10.962126 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.962014 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 01:19:10.962126 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.962048 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 01:19:10.962126 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.962064 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 01:19:10.962126 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.962073 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 01:19:10.962126 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.962099 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 01:19:10.962403 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.962164 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-bvfsl\"" Oct 30 01:19:10.973643 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:10.973620 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 01:19:11.002577 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:11.002551 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:19:11.002707 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:11.002585 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:19:11.002707 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:11.002602 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4t5n\" (UniqueName: \"kubernetes.io/projected/66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc-kube-api-access-s4t5n\") pod \"alertmanager-main-0\" (UID: \"66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:19:11.002707 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:11.002691 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc-config-out\") pod \"alertmanager-main-0\" (UID: \"66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:19:11.002818 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:11.002736 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:19:11.002818 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:11.002765 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc-config-volume\") pod \"alertmanager-main-0\" (UID: \"66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:19:11.002818 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:11.002782 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:19:11.002904 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:11.002854 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:19:11.002904 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:11.002873 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:19:11.002904 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:11.002893 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:19:11.002991 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:11.002912 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:19:11.002991 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:11.002951 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:19:11.003065 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:11.003055 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc-web-config\") pod \"alertmanager-main-0\" (UID: \"66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:19:11.103453 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:11.103370 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:19:11.103453 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:11.103408 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:19:11.103453 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:11.103438 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc-web-config\") pod \"alertmanager-main-0\" (UID: \"66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:19:11.103720 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:11.103507 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:19:11.103720 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:11.103576 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:19:11.103720 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:11.103605 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4t5n\" (UniqueName: \"kubernetes.io/projected/66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc-kube-api-access-s4t5n\") pod \"alertmanager-main-0\" (UID: \"66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:19:11.103868 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:11.103743 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc-config-out\") pod \"alertmanager-main-0\" (UID: \"66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:19:11.103868 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:11.103779 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:19:11.103868 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:11.103810 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc-config-volume\") pod \"alertmanager-main-0\" (UID: \"66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:19:11.103868 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:11.103832 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:19:11.104073 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:11.103887 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:19:11.104073 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:11.103918 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:19:11.104073 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:11.103952 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:19:11.104073 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:11.103979 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:19:11.104362 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:11.104332 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:19:11.106377 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:11.106344 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc-config-out\") pod \"alertmanager-main-0\" (UID: \"66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:19:11.106555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:11.106518 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc-web-config\") pod \"alertmanager-main-0\" (UID: \"66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:19:11.106629 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:11.106590 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:19:11.106818 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:11.106793 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc-config-volume\") pod \"alertmanager-main-0\" (UID: \"66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:19:11.106887 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:11.106832 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:19:11.106887 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:11.106859 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:19:11.106960 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:11.106901 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:19:11.107153 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:11.107137 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:19:11.107236 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:11.107219 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:19:11.108268 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:11.108244 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:19:11.123723 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:11.123703 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4t5n\" (UniqueName: \"kubernetes.io/projected/66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc-kube-api-access-s4t5n\") pod \"alertmanager-main-0\" (UID: \"66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:19:11.260254 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:11.260205 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 01:19:11.584415 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:11.584380 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 01:19:11.602220 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:19:11.602191 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66dd1a54_43b9_4141_b8bf_fd5cf5bf0ebc.slice/crio-44fad7b7586ac4cf9e2a06fc5895da80b7cc42d89048653d1dae4c6c27997fdc WatchSource:0}: Error finding container 44fad7b7586ac4cf9e2a06fc5895da80b7cc42d89048653d1dae4c6c27997fdc: Status 404 returned error can't find the container with id 44fad7b7586ac4cf9e2a06fc5895da80b7cc42d89048653d1dae4c6c27997fdc Oct 30 01:19:11.858382 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:11.858293 6221 generic.go:356] "Generic (PLEG): container finished" podID="66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc" containerID="ebbdd2fa6432e7d8943d98625c46fdfa82b3416c0cae1af075c3d42d8574820c" exitCode=0 Oct 30 01:19:11.858382 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:11.858341 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc","Type":"ContainerDied","Data":"ebbdd2fa6432e7d8943d98625c46fdfa82b3416c0cae1af075c3d42d8574820c"} Oct 30 01:19:11.858630 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:11.858389 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc","Type":"ContainerStarted","Data":"44fad7b7586ac4cf9e2a06fc5895da80b7cc42d89048653d1dae4c6c27997fdc"} Oct 30 01:19:12.544703 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:12.544666 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d205185b-fc45-4a76-9719-20b00107abf1" path="/var/lib/kubelet/pods/d205185b-fc45-4a76-9719-20b00107abf1/volumes" Oct 30 01:19:12.864311 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:12.864223 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc","Type":"ContainerStarted","Data":"56615ae4f2e4ad929e18881c4713386b9af01488dba5382445446f4c71b7612a"} Oct 30 01:19:12.864311 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:12.864310 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc","Type":"ContainerStarted","Data":"5ee4a8cd9696654dc78474eee9c643c74ac94a5c75615fd8916a76d502c98622"} Oct 30 01:19:12.864809 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:12.864336 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc","Type":"ContainerStarted","Data":"c9bbaf5fdbb12ae864ff0b832179138453e9dca5585981570809871217c5dac8"} Oct 30 01:19:12.864809 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:12.864361 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc","Type":"ContainerStarted","Data":"a85ce8ec430e9269c0d67cf9b95699cc782d8db7a448db2dc5f9f697d8d48d4e"} Oct 30 01:19:12.864809 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:12.864397 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc","Type":"ContainerStarted","Data":"157f2f51b2019c910e8ed0e3c744f4da5e924048753750aebb90bae60d7a5003"} Oct 30 01:19:12.864809 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:12.864423 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc","Type":"ContainerStarted","Data":"f9aa68b833cec5a6a7055dd3c8955fabdb16da083218a74bf74c6b9bc7de9064"} Oct 30 01:19:12.946471 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:12.946419 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.932574927 podStartE2EDuration="2.946407769s" totalImagesPullingTime="0s" totalInitContainerRuntime="13.832842ms" isStatefulPod=false podCreationTimestamp="2026-10-30 01:19:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:19:12.946282669 +0000 UTC m=+76.524570041" watchObservedRunningTime="2026-10-30 01:19:12.946407769 +0000 UTC m=+76.524695099" Oct 30 01:19:15.930070 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:15.930036 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5494f674b9-8wxhh" Oct 30 01:19:15.930493 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:15.930124 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5494f674b9-8wxhh" Oct 30 01:19:15.934762 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:15.934740 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5494f674b9-8wxhh" Oct 30 01:19:16.882174 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:16.882146 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5494f674b9-8wxhh" Oct 30 01:19:16.952747 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:16.952719 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9477b4757-pfqgz"] Oct 30 01:19:21.492793 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:19:21.492752 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:19:41.971744 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:41.971640 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-9477b4757-pfqgz" podUID="7d834c2a-f008-492e-9aac-362fe7a73672" containerName="console" containerID="cri-o://81eb36f8d4fbb810a26e5d10f9f78920fed632af86ea4dac4f5eebbc9fa52ee4" gracePeriod=15 Oct 30 01:19:42.218824 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:42.218798 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9477b4757-pfqgz" Oct 30 01:19:42.345451 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:42.345413 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d834c2a-f008-492e-9aac-362fe7a73672-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d834c2a-f008-492e-9aac-362fe7a73672-console-serving-cert\") pod \"7d834c2a-f008-492e-9aac-362fe7a73672\" (UID: \"7d834c2a-f008-492e-9aac-362fe7a73672\") " Oct 30 01:19:42.345639 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:42.345472 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d834c2a-f008-492e-9aac-362fe7a73672-tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d834c2a-f008-492e-9aac-362fe7a73672-tmp\") pod \"7d834c2a-f008-492e-9aac-362fe7a73672\" (UID: \"7d834c2a-f008-492e-9aac-362fe7a73672\") " Oct 30 01:19:42.345639 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:42.345510 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d834c2a-f008-492e-9aac-362fe7a73672-kube-api-access-sd9jh\" (UniqueName: \"kubernetes.io/projected/7d834c2a-f008-492e-9aac-362fe7a73672-kube-api-access-sd9jh\") pod \"7d834c2a-f008-492e-9aac-362fe7a73672\" (UID: \"7d834c2a-f008-492e-9aac-362fe7a73672\") " Oct 30 01:19:42.345639 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:42.345562 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d834c2a-f008-492e-9aac-362fe7a73672-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d834c2a-f008-492e-9aac-362fe7a73672-console-oauth-config\") pod \"7d834c2a-f008-492e-9aac-362fe7a73672\" (UID: \"7d834c2a-f008-492e-9aac-362fe7a73672\") " Oct 30 01:19:42.345639 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:42.345611 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d834c2a-f008-492e-9aac-362fe7a73672-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d834c2a-f008-492e-9aac-362fe7a73672-trusted-ca-bundle\") pod \"7d834c2a-f008-492e-9aac-362fe7a73672\" (UID: \"7d834c2a-f008-492e-9aac-362fe7a73672\") " Oct 30 01:19:42.345851 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:42.345649 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d834c2a-f008-492e-9aac-362fe7a73672-console-config\" (UniqueName: \"kubernetes.io/configmap/7d834c2a-f008-492e-9aac-362fe7a73672-console-config\") pod \"7d834c2a-f008-492e-9aac-362fe7a73672\" (UID: \"7d834c2a-f008-492e-9aac-362fe7a73672\") " Oct 30 01:19:42.345851 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:42.345683 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d834c2a-f008-492e-9aac-362fe7a73672-service-ca\" (UniqueName: \"kubernetes.io/configmap/7d834c2a-f008-492e-9aac-362fe7a73672-service-ca\") pod \"7d834c2a-f008-492e-9aac-362fe7a73672\" (UID: \"7d834c2a-f008-492e-9aac-362fe7a73672\") " Oct 30 01:19:42.345851 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:42.345708 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d834c2a-f008-492e-9aac-362fe7a73672-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d834c2a-f008-492e-9aac-362fe7a73672-oauth-serving-cert\") pod \"7d834c2a-f008-492e-9aac-362fe7a73672\" (UID: \"7d834c2a-f008-492e-9aac-362fe7a73672\") " Oct 30 01:19:42.345851 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:42.345749 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d834c2a-f008-492e-9aac-362fe7a73672-tmp" pod "7d834c2a-f008-492e-9aac-362fe7a73672" (UID: "7d834c2a-f008-492e-9aac-362fe7a73672"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:19:42.346062 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:42.345952 6221 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7d834c2a-f008-492e-9aac-362fe7a73672-tmp\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:19:42.346115 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:42.346073 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d834c2a-f008-492e-9aac-362fe7a73672-service-ca" pod "7d834c2a-f008-492e-9aac-362fe7a73672" (UID: "7d834c2a-f008-492e-9aac-362fe7a73672"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:42.346115 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:42.346095 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d834c2a-f008-492e-9aac-362fe7a73672-trusted-ca-bundle" pod "7d834c2a-f008-492e-9aac-362fe7a73672" (UID: "7d834c2a-f008-492e-9aac-362fe7a73672"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:42.346246 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:42.346108 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d834c2a-f008-492e-9aac-362fe7a73672-console-config" pod "7d834c2a-f008-492e-9aac-362fe7a73672" (UID: "7d834c2a-f008-492e-9aac-362fe7a73672"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:42.346328 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:42.346300 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d834c2a-f008-492e-9aac-362fe7a73672-oauth-serving-cert" pod "7d834c2a-f008-492e-9aac-362fe7a73672" (UID: "7d834c2a-f008-492e-9aac-362fe7a73672"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:19:42.347715 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:42.347694 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d834c2a-f008-492e-9aac-362fe7a73672-console-serving-cert" pod "7d834c2a-f008-492e-9aac-362fe7a73672" (UID: "7d834c2a-f008-492e-9aac-362fe7a73672"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:19:42.348146 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:42.348123 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d834c2a-f008-492e-9aac-362fe7a73672-kube-api-access-sd9jh" pod "7d834c2a-f008-492e-9aac-362fe7a73672" (UID: "7d834c2a-f008-492e-9aac-362fe7a73672"). InnerVolumeSpecName "kube-api-access-sd9jh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:19:42.348146 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:42.348129 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d834c2a-f008-492e-9aac-362fe7a73672-console-oauth-config" pod "7d834c2a-f008-492e-9aac-362fe7a73672" (UID: "7d834c2a-f008-492e-9aac-362fe7a73672"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:19:42.446763 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:42.446715 6221 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7d834c2a-f008-492e-9aac-362fe7a73672-service-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:19:42.446763 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:42.446757 6221 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7d834c2a-f008-492e-9aac-362fe7a73672-oauth-serving-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:19:42.446763 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:42.446768 6221 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7d834c2a-f008-492e-9aac-362fe7a73672-console-serving-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:19:42.446763 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:42.446777 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd9jh\" (UniqueName: \"kubernetes.io/projected/7d834c2a-f008-492e-9aac-362fe7a73672-kube-api-access-sd9jh\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:19:42.447025 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:42.446787 6221 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7d834c2a-f008-492e-9aac-362fe7a73672-console-oauth-config\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:19:42.447025 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:42.446795 6221 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d834c2a-f008-492e-9aac-362fe7a73672-trusted-ca-bundle\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:19:42.447025 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:42.446804 6221 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7d834c2a-f008-492e-9aac-362fe7a73672-console-config\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:19:42.959958 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:42.959920 6221 generic.go:356] "Generic (PLEG): container finished" podID="7d834c2a-f008-492e-9aac-362fe7a73672" containerID="81eb36f8d4fbb810a26e5d10f9f78920fed632af86ea4dac4f5eebbc9fa52ee4" exitCode=0 Oct 30 01:19:42.960128 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:42.959965 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9477b4757-pfqgz" Oct 30 01:19:42.960128 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:42.960005 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9477b4757-pfqgz" event={"ID":"7d834c2a-f008-492e-9aac-362fe7a73672","Type":"ContainerDied","Data":"81eb36f8d4fbb810a26e5d10f9f78920fed632af86ea4dac4f5eebbc9fa52ee4"} Oct 30 01:19:42.960128 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:42.960050 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9477b4757-pfqgz" event={"ID":"7d834c2a-f008-492e-9aac-362fe7a73672","Type":"ContainerDied","Data":"a095b5dc32b2cf300fed1b5e01fe9e5bb6ede4318878b8166dfe02367469e34c"} Oct 30 01:19:42.960128 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:42.960069 6221 scope.go:122] "RemoveContainer" containerID="81eb36f8d4fbb810a26e5d10f9f78920fed632af86ea4dac4f5eebbc9fa52ee4" Oct 30 01:19:42.967582 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:42.967559 6221 scope.go:122] "RemoveContainer" containerID="81eb36f8d4fbb810a26e5d10f9f78920fed632af86ea4dac4f5eebbc9fa52ee4" Oct 30 01:19:42.967883 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:19:42.967863 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81eb36f8d4fbb810a26e5d10f9f78920fed632af86ea4dac4f5eebbc9fa52ee4\": container with ID starting with 81eb36f8d4fbb810a26e5d10f9f78920fed632af86ea4dac4f5eebbc9fa52ee4 not found: ID does not exist" containerID="81eb36f8d4fbb810a26e5d10f9f78920fed632af86ea4dac4f5eebbc9fa52ee4" Oct 30 01:19:42.967946 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:42.967891 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81eb36f8d4fbb810a26e5d10f9f78920fed632af86ea4dac4f5eebbc9fa52ee4"} err="failed to get container status \"81eb36f8d4fbb810a26e5d10f9f78920fed632af86ea4dac4f5eebbc9fa52ee4\": rpc error: code = NotFound desc = could not find container \"81eb36f8d4fbb810a26e5d10f9f78920fed632af86ea4dac4f5eebbc9fa52ee4\": container with ID starting with 81eb36f8d4fbb810a26e5d10f9f78920fed632af86ea4dac4f5eebbc9fa52ee4 not found: ID does not exist" Oct 30 01:19:43.011118 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:43.011080 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9477b4757-pfqgz"] Oct 30 01:19:43.023438 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:43.023410 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-9477b4757-pfqgz"] Oct 30 01:19:44.544698 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:19:44.544659 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d834c2a-f008-492e-9aac-362fe7a73672" path="/var/lib/kubelet/pods/7d834c2a-f008-492e-9aac-362fe7a73672/volumes" Oct 30 01:19:51.493063 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:19:51.493019 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:20:21.492468 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:20:21.492430 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:20:51.492137 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:20:51.492100 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:21:21.492618 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:21:21.492526 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:21:22.545403 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.545364 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vmm7h"] Oct 30 01:21:22.545827 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.545677 6221 cpu_manager.go:414] "containerMap: removing container" podUID="7d834c2a-f008-492e-9aac-362fe7a73672" containerName="console" Oct 30 01:21:22.545827 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.545690 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d834c2a-f008-492e-9aac-362fe7a73672" containerName="console" Oct 30 01:21:22.545827 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.545750 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d834c2a-f008-492e-9aac-362fe7a73672" containerName="console" Oct 30 01:21:22.548458 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.548444 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vmm7h" Oct 30 01:21:22.578002 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.577977 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 01:21:22.578002 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.577986 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 01:21:22.578426 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.578406 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vmm7h"] Oct 30 01:21:22.628976 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.628941 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-kz2qx\"" Oct 30 01:21:22.670390 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.670351 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gr2c\" (UniqueName: \"kubernetes.io/projected/5ac5f942-fc23-42d8-beca-1e36451d7abf-kube-api-access-7gr2c\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vmm7h\" (UID: \"5ac5f942-fc23-42d8-beca-1e36451d7abf\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vmm7h" Oct 30 01:21:22.670604 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.670449 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ac5f942-fc23-42d8-beca-1e36451d7abf-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vmm7h\" (UID: \"5ac5f942-fc23-42d8-beca-1e36451d7abf\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vmm7h" Oct 30 01:21:22.670604 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.670554 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5ac5f942-fc23-42d8-beca-1e36451d7abf-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vmm7h\" (UID: \"5ac5f942-fc23-42d8-beca-1e36451d7abf\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vmm7h" Oct 30 01:21:22.747796 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.747761 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-v26wx"] Oct 30 01:21:22.750897 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.750880 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v26wx" Oct 30 01:21:22.765808 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.765784 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 01:21:22.765931 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.765850 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 01:21:22.771225 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.771199 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gr2c\" (UniqueName: \"kubernetes.io/projected/5ac5f942-fc23-42d8-beca-1e36451d7abf-kube-api-access-7gr2c\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vmm7h\" (UID: \"5ac5f942-fc23-42d8-beca-1e36451d7abf\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vmm7h" Oct 30 01:21:22.771328 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.771255 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ac5f942-fc23-42d8-beca-1e36451d7abf-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vmm7h\" (UID: \"5ac5f942-fc23-42d8-beca-1e36451d7abf\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vmm7h" Oct 30 01:21:22.771328 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.771275 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5ac5f942-fc23-42d8-beca-1e36451d7abf-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vmm7h\" (UID: \"5ac5f942-fc23-42d8-beca-1e36451d7abf\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vmm7h" Oct 30 01:21:22.771631 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.771614 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5ac5f942-fc23-42d8-beca-1e36451d7abf-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vmm7h\" (UID: \"5ac5f942-fc23-42d8-beca-1e36451d7abf\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vmm7h" Oct 30 01:21:22.771702 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.771631 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ac5f942-fc23-42d8-beca-1e36451d7abf-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vmm7h\" (UID: \"5ac5f942-fc23-42d8-beca-1e36451d7abf\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vmm7h" Oct 30 01:21:22.773231 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.773208 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-v26wx"] Oct 30 01:21:22.791236 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.791206 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gr2c\" (UniqueName: \"kubernetes.io/projected/5ac5f942-fc23-42d8-beca-1e36451d7abf-kube-api-access-7gr2c\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vmm7h\" (UID: \"5ac5f942-fc23-42d8-beca-1e36451d7abf\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vmm7h" Oct 30 01:21:22.829631 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.829548 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 01:21:22.829631 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.829527 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 01:21:22.829810 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.829755 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-44rm5\"" Oct 30 01:21:22.863953 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.863926 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vmm7h" Oct 30 01:21:22.865688 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.865673 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 01:21:22.865746 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.865686 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 01:21:22.872105 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.872085 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1665b186-6b0c-4cf4-8505-261e21f56cbf-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-v26wx\" (UID: \"1665b186-6b0c-4cf4-8505-261e21f56cbf\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v26wx" Oct 30 01:21:22.872191 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.872139 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snkln\" (UniqueName: \"kubernetes.io/projected/1665b186-6b0c-4cf4-8505-261e21f56cbf-kube-api-access-snkln\") pod \"istiod-openshift-gateway-7876bfcb74-v26wx\" (UID: \"1665b186-6b0c-4cf4-8505-261e21f56cbf\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v26wx" Oct 30 01:21:22.872265 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.872242 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1665b186-6b0c-4cf4-8505-261e21f56cbf-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-v26wx\" (UID: \"1665b186-6b0c-4cf4-8505-261e21f56cbf\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v26wx" Oct 30 01:21:22.872387 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.872369 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1665b186-6b0c-4cf4-8505-261e21f56cbf-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-v26wx\" (UID: \"1665b186-6b0c-4cf4-8505-261e21f56cbf\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v26wx" Oct 30 01:21:22.872429 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.872396 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1665b186-6b0c-4cf4-8505-261e21f56cbf-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-v26wx\" (UID: \"1665b186-6b0c-4cf4-8505-261e21f56cbf\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v26wx" Oct 30 01:21:22.872429 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.872421 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1665b186-6b0c-4cf4-8505-261e21f56cbf-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-v26wx\" (UID: \"1665b186-6b0c-4cf4-8505-261e21f56cbf\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v26wx" Oct 30 01:21:22.872565 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.872546 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1665b186-6b0c-4cf4-8505-261e21f56cbf-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-v26wx\" (UID: \"1665b186-6b0c-4cf4-8505-261e21f56cbf\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v26wx" Oct 30 01:21:22.973251 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.973212 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snkln\" (UniqueName: \"kubernetes.io/projected/1665b186-6b0c-4cf4-8505-261e21f56cbf-kube-api-access-snkln\") pod \"istiod-openshift-gateway-7876bfcb74-v26wx\" (UID: \"1665b186-6b0c-4cf4-8505-261e21f56cbf\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v26wx" Oct 30 01:21:22.973251 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.973252 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1665b186-6b0c-4cf4-8505-261e21f56cbf-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-v26wx\" (UID: \"1665b186-6b0c-4cf4-8505-261e21f56cbf\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v26wx" Oct 30 01:21:22.973507 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.973306 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1665b186-6b0c-4cf4-8505-261e21f56cbf-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-v26wx\" (UID: \"1665b186-6b0c-4cf4-8505-261e21f56cbf\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v26wx" Oct 30 01:21:22.973507 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.973328 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1665b186-6b0c-4cf4-8505-261e21f56cbf-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-v26wx\" (UID: \"1665b186-6b0c-4cf4-8505-261e21f56cbf\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v26wx" Oct 30 01:21:22.973507 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.973359 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1665b186-6b0c-4cf4-8505-261e21f56cbf-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-v26wx\" (UID: \"1665b186-6b0c-4cf4-8505-261e21f56cbf\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v26wx" Oct 30 01:21:22.973507 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.973393 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1665b186-6b0c-4cf4-8505-261e21f56cbf-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-v26wx\" (UID: \"1665b186-6b0c-4cf4-8505-261e21f56cbf\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v26wx" Oct 30 01:21:22.973507 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.973427 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1665b186-6b0c-4cf4-8505-261e21f56cbf-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-v26wx\" (UID: \"1665b186-6b0c-4cf4-8505-261e21f56cbf\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v26wx" Oct 30 01:21:22.974066 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.974040 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1665b186-6b0c-4cf4-8505-261e21f56cbf-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-v26wx\" (UID: \"1665b186-6b0c-4cf4-8505-261e21f56cbf\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v26wx" Oct 30 01:21:22.975651 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.975619 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1665b186-6b0c-4cf4-8505-261e21f56cbf-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-v26wx\" (UID: \"1665b186-6b0c-4cf4-8505-261e21f56cbf\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v26wx" Oct 30 01:21:22.975804 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.975784 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1665b186-6b0c-4cf4-8505-261e21f56cbf-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-v26wx\" (UID: \"1665b186-6b0c-4cf4-8505-261e21f56cbf\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v26wx" Oct 30 01:21:22.975864 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.975788 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1665b186-6b0c-4cf4-8505-261e21f56cbf-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-v26wx\" (UID: \"1665b186-6b0c-4cf4-8505-261e21f56cbf\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v26wx" Oct 30 01:21:22.975864 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.975819 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1665b186-6b0c-4cf4-8505-261e21f56cbf-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-v26wx\" (UID: \"1665b186-6b0c-4cf4-8505-261e21f56cbf\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v26wx" Oct 30 01:21:22.993124 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.993094 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snkln\" (UniqueName: \"kubernetes.io/projected/1665b186-6b0c-4cf4-8505-261e21f56cbf-kube-api-access-snkln\") pod \"istiod-openshift-gateway-7876bfcb74-v26wx\" (UID: \"1665b186-6b0c-4cf4-8505-261e21f56cbf\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v26wx" Oct 30 01:21:22.993124 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:22.993113 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1665b186-6b0c-4cf4-8505-261e21f56cbf-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-v26wx\" (UID: \"1665b186-6b0c-4cf4-8505-261e21f56cbf\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v26wx" Oct 30 01:21:23.061646 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:23.061610 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v26wx" Oct 30 01:21:23.159882 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:23.159849 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vmm7h"] Oct 30 01:21:23.163997 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:21:23.163971 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ac5f942_fc23_42d8_beca_1e36451d7abf.slice/crio-a9ab3e60a1e4ff9882252a1e938ca0d6bdecb92a0fcdc824ba07c1a5aa60cc86 WatchSource:0}: Error finding container a9ab3e60a1e4ff9882252a1e938ca0d6bdecb92a0fcdc824ba07c1a5aa60cc86: Status 404 returned error can't find the container with id a9ab3e60a1e4ff9882252a1e938ca0d6bdecb92a0fcdc824ba07c1a5aa60cc86 Oct 30 01:21:23.165811 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:23.165795 6221 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:21:23.273728 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:23.273693 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vmm7h" event={"ID":"5ac5f942-fc23-42d8-beca-1e36451d7abf","Type":"ContainerStarted","Data":"a9ab3e60a1e4ff9882252a1e938ca0d6bdecb92a0fcdc824ba07c1a5aa60cc86"} Oct 30 01:21:23.357203 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:23.357114 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-v26wx"] Oct 30 01:21:23.361118 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:21:23.361091 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1665b186_6b0c_4cf4_8505_261e21f56cbf.slice/crio-dc93afaed1bd39230835fb352996abdb3e891b33e124de860ed264d60084c257 WatchSource:0}: Error finding container dc93afaed1bd39230835fb352996abdb3e891b33e124de860ed264d60084c257: Status 404 returned error can't find the container with id dc93afaed1bd39230835fb352996abdb3e891b33e124de860ed264d60084c257 Oct 30 01:21:24.281222 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:24.281181 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v26wx" event={"ID":"1665b186-6b0c-4cf4-8505-261e21f56cbf","Type":"ContainerStarted","Data":"dc93afaed1bd39230835fb352996abdb3e891b33e124de860ed264d60084c257"} Oct 30 01:21:26.221301 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:26.221257 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:21:26.221678 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:26.221344 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:21:26.290704 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:26.290673 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v26wx" event={"ID":"1665b186-6b0c-4cf4-8505-261e21f56cbf","Type":"ContainerStarted","Data":"90c754399db642fd82a252e189a536e826ec5fbda048c03e11dd80fd9b5f0b13"} Oct 30 01:21:26.290845 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:26.290776 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v26wx" Oct 30 01:21:26.336843 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:26.336791 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v26wx" podStartSLOduration=1.479524149 podStartE2EDuration="4.336777547s" totalImagesPullingTime="2.857253398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:21:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:21:26.336297358 +0000 UTC m=+209.914584696" watchObservedRunningTime="2026-10-30 01:21:26.336777547 +0000 UTC m=+209.915064874" Oct 30 01:21:27.296174 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:27.296128 6221 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-v26wx container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:21:27.296755 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:27.296202 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v26wx" podUID="1665b186-6b0c-4cf4-8505-261e21f56cbf" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:21:29.303214 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:29.303175 6221 generic.go:356] "Generic (PLEG): container finished" podID="5ac5f942-fc23-42d8-beca-1e36451d7abf" containerID="2849c786a0dddddb9ae0dc28be282a18b708b62c1a7d23ecb308e4919264ca07" exitCode=0 Oct 30 01:21:29.303735 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:29.303221 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vmm7h" event={"ID":"5ac5f942-fc23-42d8-beca-1e36451d7abf","Type":"ContainerDied","Data":"2849c786a0dddddb9ae0dc28be282a18b708b62c1a7d23ecb308e4919264ca07"} Oct 30 01:21:30.295710 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:30.295682 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v26wx" Oct 30 01:21:30.308435 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:30.308403 6221 generic.go:356] "Generic (PLEG): container finished" podID="5ac5f942-fc23-42d8-beca-1e36451d7abf" containerID="4276b0b68c004c6112308f4a2b93c0f5392728c5c246829e956cdff8bf769a1e" exitCode=0 Oct 30 01:21:30.308809 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:30.308442 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vmm7h" event={"ID":"5ac5f942-fc23-42d8-beca-1e36451d7abf","Type":"ContainerDied","Data":"4276b0b68c004c6112308f4a2b93c0f5392728c5c246829e956cdff8bf769a1e"} Oct 30 01:21:36.336926 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:36.336860 6221 generic.go:356] "Generic (PLEG): container finished" podID="5ac5f942-fc23-42d8-beca-1e36451d7abf" containerID="26207c6f6013482e748f42b313ef3748a48b61397df4454b910390dc348d957d" exitCode=0 Oct 30 01:21:36.336926 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:36.336896 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vmm7h" event={"ID":"5ac5f942-fc23-42d8-beca-1e36451d7abf","Type":"ContainerDied","Data":"26207c6f6013482e748f42b313ef3748a48b61397df4454b910390dc348d957d"} Oct 30 01:21:37.456762 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:37.456736 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vmm7h" Oct 30 01:21:37.486322 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:37.486298 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ac5f942-fc23-42d8-beca-1e36451d7abf-kube-api-access-7gr2c\" (UniqueName: \"kubernetes.io/projected/5ac5f942-fc23-42d8-beca-1e36451d7abf-kube-api-access-7gr2c\") pod \"5ac5f942-fc23-42d8-beca-1e36451d7abf\" (UID: \"5ac5f942-fc23-42d8-beca-1e36451d7abf\") " Oct 30 01:21:37.486431 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:37.486368 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ac5f942-fc23-42d8-beca-1e36451d7abf-util\" (UniqueName: \"kubernetes.io/empty-dir/5ac5f942-fc23-42d8-beca-1e36451d7abf-util\") pod \"5ac5f942-fc23-42d8-beca-1e36451d7abf\" (UID: \"5ac5f942-fc23-42d8-beca-1e36451d7abf\") " Oct 30 01:21:37.486477 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:37.486468 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ac5f942-fc23-42d8-beca-1e36451d7abf-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ac5f942-fc23-42d8-beca-1e36451d7abf-bundle\") pod \"5ac5f942-fc23-42d8-beca-1e36451d7abf\" (UID: \"5ac5f942-fc23-42d8-beca-1e36451d7abf\") " Oct 30 01:21:37.487016 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:37.486993 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ac5f942-fc23-42d8-beca-1e36451d7abf-bundle" pod "5ac5f942-fc23-42d8-beca-1e36451d7abf" (UID: "5ac5f942-fc23-42d8-beca-1e36451d7abf"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:37.488374 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:37.488357 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ac5f942-fc23-42d8-beca-1e36451d7abf-kube-api-access-7gr2c" pod "5ac5f942-fc23-42d8-beca-1e36451d7abf" (UID: "5ac5f942-fc23-42d8-beca-1e36451d7abf"). InnerVolumeSpecName "kube-api-access-7gr2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:21:37.492050 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:37.492025 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ac5f942-fc23-42d8-beca-1e36451d7abf-util" pod "5ac5f942-fc23-42d8-beca-1e36451d7abf" (UID: "5ac5f942-fc23-42d8-beca-1e36451d7abf"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:21:37.587390 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:37.587340 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gr2c\" (UniqueName: \"kubernetes.io/projected/5ac5f942-fc23-42d8-beca-1e36451d7abf-kube-api-access-7gr2c\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:21:37.587390 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:37.587386 6221 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5ac5f942-fc23-42d8-beca-1e36451d7abf-util\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:21:37.587390 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:37.587402 6221 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5ac5f942-fc23-42d8-beca-1e36451d7abf-bundle\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:21:38.345294 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:38.345260 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vmm7h" event={"ID":"5ac5f942-fc23-42d8-beca-1e36451d7abf","Type":"ContainerDied","Data":"a9ab3e60a1e4ff9882252a1e938ca0d6bdecb92a0fcdc824ba07c1a5aa60cc86"} Oct 30 01:21:38.345480 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:38.345312 6221 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9ab3e60a1e4ff9882252a1e938ca0d6bdecb92a0fcdc824ba07c1a5aa60cc86" Oct 30 01:21:38.345480 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:21:38.345273 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71vmm7h" Oct 30 01:21:51.492423 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:21:51.492380 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:22:11.752258 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:11.752227 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-dnzc5"] Oct 30 01:22:11.752776 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:11.752476 6221 cpu_manager.go:414] "containerMap: removing container" podUID="5ac5f942-fc23-42d8-beca-1e36451d7abf" containerName="util" Oct 30 01:22:11.752776 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:11.752486 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ac5f942-fc23-42d8-beca-1e36451d7abf" containerName="util" Oct 30 01:22:11.752776 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:11.752506 6221 cpu_manager.go:414] "containerMap: removing container" podUID="5ac5f942-fc23-42d8-beca-1e36451d7abf" containerName="extract" Oct 30 01:22:11.752776 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:11.752512 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ac5f942-fc23-42d8-beca-1e36451d7abf" containerName="extract" Oct 30 01:22:11.752776 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:11.752522 6221 cpu_manager.go:414] "containerMap: removing container" podUID="5ac5f942-fc23-42d8-beca-1e36451d7abf" containerName="pull" Oct 30 01:22:11.752776 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:11.752542 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ac5f942-fc23-42d8-beca-1e36451d7abf" containerName="pull" Oct 30 01:22:11.752776 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:11.752587 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ac5f942-fc23-42d8-beca-1e36451d7abf" containerName="extract" Oct 30 01:22:11.754362 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:11.754343 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-dnzc5" Oct 30 01:22:11.790468 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:11.790428 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-dnzc5"] Oct 30 01:22:11.845318 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:11.845281 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-pp4hc\"" Oct 30 01:22:11.845318 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:11.845288 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 01:22:11.845318 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:11.845289 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 01:22:11.845628 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:11.845326 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 01:22:11.845628 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:11.845441 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 01:22:11.845628 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:11.845449 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 01:22:11.852838 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:11.852813 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e2047ee7-6511-4aa0-9f43-bade063b34e3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-685895cf4f-dnzc5\" (UID: \"e2047ee7-6511-4aa0-9f43-bade063b34e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-dnzc5" Oct 30 01:22:11.852973 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:11.852848 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e2047ee7-6511-4aa0-9f43-bade063b34e3-manifests\") pod \"opendatahub-operator-controller-manager-685895cf4f-dnzc5\" (UID: \"e2047ee7-6511-4aa0-9f43-bade063b34e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-dnzc5" Oct 30 01:22:11.852973 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:11.852880 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2047ee7-6511-4aa0-9f43-bade063b34e3-tmp\") pod \"opendatahub-operator-controller-manager-685895cf4f-dnzc5\" (UID: \"e2047ee7-6511-4aa0-9f43-bade063b34e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-dnzc5" Oct 30 01:22:11.853043 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:11.852989 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e2047ee7-6511-4aa0-9f43-bade063b34e3-webhook-cert\") pod \"opendatahub-operator-controller-manager-685895cf4f-dnzc5\" (UID: \"e2047ee7-6511-4aa0-9f43-bade063b34e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-dnzc5" Oct 30 01:22:11.853043 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:11.853020 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kz48g\" (UniqueName: \"kubernetes.io/projected/e2047ee7-6511-4aa0-9f43-bade063b34e3-kube-api-access-kz48g\") pod \"opendatahub-operator-controller-manager-685895cf4f-dnzc5\" (UID: \"e2047ee7-6511-4aa0-9f43-bade063b34e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-dnzc5" Oct 30 01:22:11.853114 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:11.853084 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e2047ee7-6511-4aa0-9f43-bade063b34e3-metrics-cert\") pod \"opendatahub-operator-controller-manager-685895cf4f-dnzc5\" (UID: \"e2047ee7-6511-4aa0-9f43-bade063b34e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-dnzc5" Oct 30 01:22:11.953704 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:11.953667 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e2047ee7-6511-4aa0-9f43-bade063b34e3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-685895cf4f-dnzc5\" (UID: \"e2047ee7-6511-4aa0-9f43-bade063b34e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-dnzc5" Oct 30 01:22:11.953704 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:11.953704 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e2047ee7-6511-4aa0-9f43-bade063b34e3-manifests\") pod \"opendatahub-operator-controller-manager-685895cf4f-dnzc5\" (UID: \"e2047ee7-6511-4aa0-9f43-bade063b34e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-dnzc5" Oct 30 01:22:11.953908 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:11.953731 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2047ee7-6511-4aa0-9f43-bade063b34e3-tmp\") pod \"opendatahub-operator-controller-manager-685895cf4f-dnzc5\" (UID: \"e2047ee7-6511-4aa0-9f43-bade063b34e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-dnzc5" Oct 30 01:22:11.953908 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:11.953750 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e2047ee7-6511-4aa0-9f43-bade063b34e3-webhook-cert\") pod \"opendatahub-operator-controller-manager-685895cf4f-dnzc5\" (UID: \"e2047ee7-6511-4aa0-9f43-bade063b34e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-dnzc5" Oct 30 01:22:11.953908 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:11.953769 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kz48g\" (UniqueName: \"kubernetes.io/projected/e2047ee7-6511-4aa0-9f43-bade063b34e3-kube-api-access-kz48g\") pod \"opendatahub-operator-controller-manager-685895cf4f-dnzc5\" (UID: \"e2047ee7-6511-4aa0-9f43-bade063b34e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-dnzc5" Oct 30 01:22:11.953908 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:11.953806 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e2047ee7-6511-4aa0-9f43-bade063b34e3-metrics-cert\") pod \"opendatahub-operator-controller-manager-685895cf4f-dnzc5\" (UID: \"e2047ee7-6511-4aa0-9f43-bade063b34e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-dnzc5" Oct 30 01:22:11.954186 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:11.954167 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e2047ee7-6511-4aa0-9f43-bade063b34e3-manifests\") pod \"opendatahub-operator-controller-manager-685895cf4f-dnzc5\" (UID: \"e2047ee7-6511-4aa0-9f43-bade063b34e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-dnzc5" Oct 30 01:22:11.956100 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:11.956072 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e2047ee7-6511-4aa0-9f43-bade063b34e3-tmp\") pod \"opendatahub-operator-controller-manager-685895cf4f-dnzc5\" (UID: \"e2047ee7-6511-4aa0-9f43-bade063b34e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-dnzc5" Oct 30 01:22:11.956233 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:11.956220 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e2047ee7-6511-4aa0-9f43-bade063b34e3-webhook-cert\") pod \"opendatahub-operator-controller-manager-685895cf4f-dnzc5\" (UID: \"e2047ee7-6511-4aa0-9f43-bade063b34e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-dnzc5" Oct 30 01:22:11.956316 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:11.956298 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e2047ee7-6511-4aa0-9f43-bade063b34e3-apiservice-cert\") pod \"opendatahub-operator-controller-manager-685895cf4f-dnzc5\" (UID: \"e2047ee7-6511-4aa0-9f43-bade063b34e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-dnzc5" Oct 30 01:22:11.956372 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:11.956322 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e2047ee7-6511-4aa0-9f43-bade063b34e3-metrics-cert\") pod \"opendatahub-operator-controller-manager-685895cf4f-dnzc5\" (UID: \"e2047ee7-6511-4aa0-9f43-bade063b34e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-dnzc5" Oct 30 01:22:11.973609 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:11.973582 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kz48g\" (UniqueName: \"kubernetes.io/projected/e2047ee7-6511-4aa0-9f43-bade063b34e3-kube-api-access-kz48g\") pod \"opendatahub-operator-controller-manager-685895cf4f-dnzc5\" (UID: \"e2047ee7-6511-4aa0-9f43-bade063b34e3\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-dnzc5" Oct 30 01:22:12.071061 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:12.071029 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-dnzc5" Oct 30 01:22:12.429563 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:12.429410 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-dnzc5"] Oct 30 01:22:12.434553 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:22:12.434514 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2047ee7_6511_4aa0_9f43_bade063b34e3.slice/crio-2d60c50fa0267c7fddf56f5641c0163d211b377b48259d0fc73557450df1952b WatchSource:0}: Error finding container 2d60c50fa0267c7fddf56f5641c0163d211b377b48259d0fc73557450df1952b: Status 404 returned error can't find the container with id 2d60c50fa0267c7fddf56f5641c0163d211b377b48259d0fc73557450df1952b Oct 30 01:22:12.458470 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:12.458438 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-dnzc5" event={"ID":"e2047ee7-6511-4aa0-9f43-bade063b34e3","Type":"ContainerStarted","Data":"2d60c50fa0267c7fddf56f5641c0163d211b377b48259d0fc73557450df1952b"} Oct 30 01:22:15.472596 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:15.472486 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-dnzc5" event={"ID":"e2047ee7-6511-4aa0-9f43-bade063b34e3","Type":"ContainerStarted","Data":"ad7ca0c6634c250be62077d01fa9fe398bb714444665e0135fae72f6cf05bbab"} Oct 30 01:22:16.477259 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:16.477222 6221 generic.go:356] "Generic (PLEG): container finished" podID="e2047ee7-6511-4aa0-9f43-bade063b34e3" containerID="ad7ca0c6634c250be62077d01fa9fe398bb714444665e0135fae72f6cf05bbab" exitCode=0 Oct 30 01:22:16.477745 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:16.477307 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-dnzc5" event={"ID":"e2047ee7-6511-4aa0-9f43-bade063b34e3","Type":"ContainerDied","Data":"ad7ca0c6634c250be62077d01fa9fe398bb714444665e0135fae72f6cf05bbab"} Oct 30 01:22:17.483057 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:17.483023 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-dnzc5" event={"ID":"e2047ee7-6511-4aa0-9f43-bade063b34e3","Type":"ContainerStarted","Data":"eacaf12ee4ab9fcc2dcf5e42cc727c2d7adcd45b8f2691acf2c00a312b1f7fef"} Oct 30 01:22:17.483444 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:17.483250 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-dnzc5" Oct 30 01:22:17.559847 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:17.559799 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-dnzc5" podStartSLOduration=3.003921497 podStartE2EDuration="6.559787339s" totalImagesPullingTime="3.062319568s" totalInitContainerRuntime="493.546274ms" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:17.547160953 +0000 UTC m=+261.125448289" watchObservedRunningTime="2026-10-30 01:22:17.559787339 +0000 UTC m=+261.138074674" Oct 30 01:22:21.493111 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:22:21.493066 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:22:28.489564 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:28.489518 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-685895cf4f-dnzc5" Oct 30 01:22:44.212867 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:44.212784 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-jlc4k"] Oct 30 01:22:44.214891 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:44.214872 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-jlc4k" Oct 30 01:22:44.233256 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:44.233227 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-jlc4k"] Oct 30 01:22:44.244690 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:44.244668 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-7k6tk\"" Oct 30 01:22:44.244690 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:44.244669 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 01:22:44.244835 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:44.244702 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 01:22:44.316355 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:44.316321 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58szz\" (UniqueName: \"kubernetes.io/projected/44370985-a465-47ea-9c78-29548b848d3c-kube-api-access-58szz\") pod \"cert-manager-597b95cb84-jlc4k\" (UID: \"44370985-a465-47ea-9c78-29548b848d3c\") " pod="cert-manager/cert-manager-597b95cb84-jlc4k" Oct 30 01:22:44.316523 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:44.316361 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/44370985-a465-47ea-9c78-29548b848d3c-bound-sa-token\") pod \"cert-manager-597b95cb84-jlc4k\" (UID: \"44370985-a465-47ea-9c78-29548b848d3c\") " pod="cert-manager/cert-manager-597b95cb84-jlc4k" Oct 30 01:22:44.416720 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:44.416685 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58szz\" (UniqueName: \"kubernetes.io/projected/44370985-a465-47ea-9c78-29548b848d3c-kube-api-access-58szz\") pod \"cert-manager-597b95cb84-jlc4k\" (UID: \"44370985-a465-47ea-9c78-29548b848d3c\") " pod="cert-manager/cert-manager-597b95cb84-jlc4k" Oct 30 01:22:44.416720 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:44.416725 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/44370985-a465-47ea-9c78-29548b848d3c-bound-sa-token\") pod \"cert-manager-597b95cb84-jlc4k\" (UID: \"44370985-a465-47ea-9c78-29548b848d3c\") " pod="cert-manager/cert-manager-597b95cb84-jlc4k" Oct 30 01:22:44.438462 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:44.438436 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/44370985-a465-47ea-9c78-29548b848d3c-bound-sa-token\") pod \"cert-manager-597b95cb84-jlc4k\" (UID: \"44370985-a465-47ea-9c78-29548b848d3c\") " pod="cert-manager/cert-manager-597b95cb84-jlc4k" Oct 30 01:22:44.438615 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:44.438585 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58szz\" (UniqueName: \"kubernetes.io/projected/44370985-a465-47ea-9c78-29548b848d3c-kube-api-access-58szz\") pod \"cert-manager-597b95cb84-jlc4k\" (UID: \"44370985-a465-47ea-9c78-29548b848d3c\") " pod="cert-manager/cert-manager-597b95cb84-jlc4k" Oct 30 01:22:44.544204 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:44.544177 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-jlc4k" Oct 30 01:22:44.855769 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:44.855694 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-jlc4k"] Oct 30 01:22:44.859174 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:22:44.859143 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44370985_a465_47ea_9c78_29548b848d3c.slice/crio-9edf8281fbdb103f4e25d9e1d97ffb4191289c735a2e3e27e65708f4c039dcd8 WatchSource:0}: Error finding container 9edf8281fbdb103f4e25d9e1d97ffb4191289c735a2e3e27e65708f4c039dcd8: Status 404 returned error can't find the container with id 9edf8281fbdb103f4e25d9e1d97ffb4191289c735a2e3e27e65708f4c039dcd8 Oct 30 01:22:45.579004 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:45.578960 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-jlc4k" event={"ID":"44370985-a465-47ea-9c78-29548b848d3c","Type":"ContainerStarted","Data":"9edf8281fbdb103f4e25d9e1d97ffb4191289c735a2e3e27e65708f4c039dcd8"} Oct 30 01:22:47.152077 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:47.152043 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-66f46d7b58-rcnz2"] Oct 30 01:22:47.154614 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:47.154592 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-66f46d7b58-rcnz2" Oct 30 01:22:47.182563 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:47.182506 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-66f46d7b58-rcnz2"] Oct 30 01:22:47.183878 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:47.183853 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 01:22:47.188106 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:47.188083 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-c6x4g\"" Oct 30 01:22:47.188223 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:47.188130 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 01:22:47.244236 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:47.244208 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gxx8\" (UniqueName: \"kubernetes.io/projected/6852625d-7a8f-42fe-a064-b811802f5daa-kube-api-access-7gxx8\") pod \"kube-auth-proxy-66f46d7b58-rcnz2\" (UID: \"6852625d-7a8f-42fe-a064-b811802f5daa\") " pod="openshift-ingress/kube-auth-proxy-66f46d7b58-rcnz2" Oct 30 01:22:47.244396 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:47.244258 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6852625d-7a8f-42fe-a064-b811802f5daa-tls-certs\") pod \"kube-auth-proxy-66f46d7b58-rcnz2\" (UID: \"6852625d-7a8f-42fe-a064-b811802f5daa\") " pod="openshift-ingress/kube-auth-proxy-66f46d7b58-rcnz2" Oct 30 01:22:47.244396 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:47.244356 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6852625d-7a8f-42fe-a064-b811802f5daa-tmp\") pod \"kube-auth-proxy-66f46d7b58-rcnz2\" (UID: \"6852625d-7a8f-42fe-a064-b811802f5daa\") " pod="openshift-ingress/kube-auth-proxy-66f46d7b58-rcnz2" Oct 30 01:22:47.345277 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:47.345241 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gxx8\" (UniqueName: \"kubernetes.io/projected/6852625d-7a8f-42fe-a064-b811802f5daa-kube-api-access-7gxx8\") pod \"kube-auth-proxy-66f46d7b58-rcnz2\" (UID: \"6852625d-7a8f-42fe-a064-b811802f5daa\") " pod="openshift-ingress/kube-auth-proxy-66f46d7b58-rcnz2" Oct 30 01:22:47.345277 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:47.345288 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6852625d-7a8f-42fe-a064-b811802f5daa-tls-certs\") pod \"kube-auth-proxy-66f46d7b58-rcnz2\" (UID: \"6852625d-7a8f-42fe-a064-b811802f5daa\") " pod="openshift-ingress/kube-auth-proxy-66f46d7b58-rcnz2" Oct 30 01:22:47.345514 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:47.345322 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6852625d-7a8f-42fe-a064-b811802f5daa-tmp\") pod \"kube-auth-proxy-66f46d7b58-rcnz2\" (UID: \"6852625d-7a8f-42fe-a064-b811802f5daa\") " pod="openshift-ingress/kube-auth-proxy-66f46d7b58-rcnz2" Oct 30 01:22:47.347711 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:47.347683 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6852625d-7a8f-42fe-a064-b811802f5daa-tmp\") pod \"kube-auth-proxy-66f46d7b58-rcnz2\" (UID: \"6852625d-7a8f-42fe-a064-b811802f5daa\") " pod="openshift-ingress/kube-auth-proxy-66f46d7b58-rcnz2" Oct 30 01:22:47.348058 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:47.348037 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6852625d-7a8f-42fe-a064-b811802f5daa-tls-certs\") pod \"kube-auth-proxy-66f46d7b58-rcnz2\" (UID: \"6852625d-7a8f-42fe-a064-b811802f5daa\") " pod="openshift-ingress/kube-auth-proxy-66f46d7b58-rcnz2" Oct 30 01:22:47.372972 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:47.372940 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gxx8\" (UniqueName: \"kubernetes.io/projected/6852625d-7a8f-42fe-a064-b811802f5daa-kube-api-access-7gxx8\") pod \"kube-auth-proxy-66f46d7b58-rcnz2\" (UID: \"6852625d-7a8f-42fe-a064-b811802f5daa\") " pod="openshift-ingress/kube-auth-proxy-66f46d7b58-rcnz2" Oct 30 01:22:47.468247 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:47.468169 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-66f46d7b58-rcnz2" Oct 30 01:22:47.770248 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:47.770174 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-66f46d7b58-rcnz2"] Oct 30 01:22:47.774446 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:22:47.774415 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6852625d_7a8f_42fe_a064_b811802f5daa.slice/crio-21321b1f4ad66c095253dd7a125995e2856f02970979fbac1d53b568d96be687 WatchSource:0}: Error finding container 21321b1f4ad66c095253dd7a125995e2856f02970979fbac1d53b568d96be687: Status 404 returned error can't find the container with id 21321b1f4ad66c095253dd7a125995e2856f02970979fbac1d53b568d96be687 Oct 30 01:22:48.591236 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:48.591196 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-66f46d7b58-rcnz2" event={"ID":"6852625d-7a8f-42fe-a064-b811802f5daa","Type":"ContainerStarted","Data":"21321b1f4ad66c095253dd7a125995e2856f02970979fbac1d53b568d96be687"} Oct 30 01:22:48.593179 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:48.593147 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-jlc4k" event={"ID":"44370985-a465-47ea-9c78-29548b848d3c","Type":"ContainerStarted","Data":"4dc9add55412f94123c02bc0235faeabd8dd6c84078488d1f525ec21395a6bbc"} Oct 30 01:22:48.633108 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:48.633059 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-jlc4k" podStartSLOduration=1.8354892550000002 podStartE2EDuration="4.633045745s" totalImagesPullingTime="2.79755649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:48.632296843 +0000 UTC m=+292.210584182" watchObservedRunningTime="2026-10-30 01:22:48.633045745 +0000 UTC m=+292.211333077" Oct 30 01:22:50.602604 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:50.602567 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-66f46d7b58-rcnz2" event={"ID":"6852625d-7a8f-42fe-a064-b811802f5daa","Type":"ContainerStarted","Data":"66ee05e90abf96d9f61ed0d869515d5ce047adec0cf439d74ce2973c185ec60e"} Oct 30 01:22:50.648747 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:50.648696 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-66f46d7b58-rcnz2" podStartSLOduration=1.304754085 podStartE2EDuration="3.64868362s" totalImagesPullingTime="2.343929535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:22:50.647702497 +0000 UTC m=+294.225989834" watchObservedRunningTime="2026-10-30 01:22:50.64868362 +0000 UTC m=+294.226970956" Oct 30 01:22:51.491421 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:22:51.491381 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:22:56.569479 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:56.569452 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2vsns_d332bbb4-c699-44e8-96ba-516d8ca03e95/ovn-acl-logging/0.log" Oct 30 01:22:56.570022 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:22:56.569452 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2vsns_d332bbb4-c699-44e8-96ba-516d8ca03e95/ovn-acl-logging/0.log" Oct 30 01:23:05.362197 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:05.362160 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:23:05.364938 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:05.364918 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-67lvt" Oct 30 01:23:05.382983 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:05.382957 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:23:05.398852 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:05.398834 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 01:23:05.398852 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:05.398833 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 01:23:05.398998 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:05.398866 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-7tx6b\"" Oct 30 01:23:05.398998 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:05.398921 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 01:23:05.459285 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:05.459255 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:23:05.503975 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:05.503947 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/53979da9-49dd-440b-b647-bcc57e7f738c-config-file\") pod \"limitador-limitador-7c9b87d574-67lvt\" (UID: \"53979da9-49dd-440b-b647-bcc57e7f738c\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-67lvt" Oct 30 01:23:05.504150 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:05.504090 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h62tg\" (UniqueName: \"kubernetes.io/projected/53979da9-49dd-440b-b647-bcc57e7f738c-kube-api-access-h62tg\") pod \"limitador-limitador-7c9b87d574-67lvt\" (UID: \"53979da9-49dd-440b-b647-bcc57e7f738c\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-67lvt" Oct 30 01:23:05.604776 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:05.604736 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h62tg\" (UniqueName: \"kubernetes.io/projected/53979da9-49dd-440b-b647-bcc57e7f738c-kube-api-access-h62tg\") pod \"limitador-limitador-7c9b87d574-67lvt\" (UID: \"53979da9-49dd-440b-b647-bcc57e7f738c\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-67lvt" Oct 30 01:23:05.604776 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:05.604779 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/53979da9-49dd-440b-b647-bcc57e7f738c-config-file\") pod \"limitador-limitador-7c9b87d574-67lvt\" (UID: \"53979da9-49dd-440b-b647-bcc57e7f738c\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-67lvt" Oct 30 01:23:05.605386 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:05.605358 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/53979da9-49dd-440b-b647-bcc57e7f738c-config-file\") pod \"limitador-limitador-7c9b87d574-67lvt\" (UID: \"53979da9-49dd-440b-b647-bcc57e7f738c\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-67lvt" Oct 30 01:23:05.626206 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:05.626160 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h62tg\" (UniqueName: \"kubernetes.io/projected/53979da9-49dd-440b-b647-bcc57e7f738c-kube-api-access-h62tg\") pod \"limitador-limitador-7c9b87d574-67lvt\" (UID: \"53979da9-49dd-440b-b647-bcc57e7f738c\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-67lvt" Oct 30 01:23:05.675856 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:05.675826 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-67lvt" Oct 30 01:23:05.969321 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:05.969233 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:23:05.973076 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:23:05.973041 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53979da9_49dd_440b_b647_bcc57e7f738c.slice/crio-ee9234f7a009cdb3c366c97836cd6a5690984ac9c1e103b2ea7b8ebc21ad154f WatchSource:0}: Error finding container ee9234f7a009cdb3c366c97836cd6a5690984ac9c1e103b2ea7b8ebc21ad154f: Status 404 returned error can't find the container with id ee9234f7a009cdb3c366c97836cd6a5690984ac9c1e103b2ea7b8ebc21ad154f Oct 30 01:23:06.661214 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:06.661154 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-67lvt" event={"ID":"53979da9-49dd-440b-b647-bcc57e7f738c","Type":"ContainerStarted","Data":"ee9234f7a009cdb3c366c97836cd6a5690984ac9c1e103b2ea7b8ebc21ad154f"} Oct 30 01:23:08.670468 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:08.670426 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-67lvt" event={"ID":"53979da9-49dd-440b-b647-bcc57e7f738c","Type":"ContainerStarted","Data":"47737c10baaa8de335456e88a0aabe6e1a1d8ef9d2c538a25ceddb12f8f91830"} Oct 30 01:23:08.670879 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:08.670580 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-67lvt" Oct 30 01:23:08.715909 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:08.715861 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-67lvt" podStartSLOduration=1.541755294 podStartE2EDuration="3.715848813s" totalImagesPullingTime="2.174093519s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:08.715794103 +0000 UTC m=+312.294081439" watchObservedRunningTime="2026-10-30 01:23:08.715848813 +0000 UTC m=+312.294136148" Oct 30 01:23:19.675499 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:19.675471 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-67lvt" Oct 30 01:23:21.492316 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:23:21.492279 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:23:51.492671 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:23:51.492633 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:23:51.632715 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:51.632681 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wh5tn"] Oct 30 01:23:51.636218 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:51.636194 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wh5tn" Oct 30 01:23:51.651224 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:51.651205 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a9c11a6-3a7f-473b-92c5-80a93a69bc34-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wh5tn\" (UID: \"3a9c11a6-3a7f-473b-92c5-80a93a69bc34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wh5tn" Oct 30 01:23:51.651322 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:51.651241 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3a9c11a6-3a7f-473b-92c5-80a93a69bc34-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wh5tn\" (UID: \"3a9c11a6-3a7f-473b-92c5-80a93a69bc34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wh5tn" Oct 30 01:23:51.651322 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:51.651261 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a9c11a6-3a7f-473b-92c5-80a93a69bc34-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wh5tn\" (UID: \"3a9c11a6-3a7f-473b-92c5-80a93a69bc34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wh5tn" Oct 30 01:23:51.651398 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:51.651385 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3a9c11a6-3a7f-473b-92c5-80a93a69bc34-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wh5tn\" (UID: \"3a9c11a6-3a7f-473b-92c5-80a93a69bc34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wh5tn" Oct 30 01:23:51.651435 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:51.651407 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2xxq\" (UniqueName: \"kubernetes.io/projected/3a9c11a6-3a7f-473b-92c5-80a93a69bc34-kube-api-access-w2xxq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wh5tn\" (UID: \"3a9c11a6-3a7f-473b-92c5-80a93a69bc34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wh5tn" Oct 30 01:23:51.651511 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:51.651486 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3a9c11a6-3a7f-473b-92c5-80a93a69bc34-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wh5tn\" (UID: \"3a9c11a6-3a7f-473b-92c5-80a93a69bc34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wh5tn" Oct 30 01:23:51.651566 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:51.651546 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3a9c11a6-3a7f-473b-92c5-80a93a69bc34-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wh5tn\" (UID: \"3a9c11a6-3a7f-473b-92c5-80a93a69bc34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wh5tn" Oct 30 01:23:51.664980 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:51.664952 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wh5tn"] Oct 30 01:23:51.752973 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:51.752899 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3a9c11a6-3a7f-473b-92c5-80a93a69bc34-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wh5tn\" (UID: \"3a9c11a6-3a7f-473b-92c5-80a93a69bc34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wh5tn" Oct 30 01:23:51.752973 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:51.752929 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2xxq\" (UniqueName: \"kubernetes.io/projected/3a9c11a6-3a7f-473b-92c5-80a93a69bc34-kube-api-access-w2xxq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wh5tn\" (UID: \"3a9c11a6-3a7f-473b-92c5-80a93a69bc34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wh5tn" Oct 30 01:23:51.752973 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:51.752954 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3a9c11a6-3a7f-473b-92c5-80a93a69bc34-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wh5tn\" (UID: \"3a9c11a6-3a7f-473b-92c5-80a93a69bc34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wh5tn" Oct 30 01:23:51.753227 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:51.752985 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3a9c11a6-3a7f-473b-92c5-80a93a69bc34-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wh5tn\" (UID: \"3a9c11a6-3a7f-473b-92c5-80a93a69bc34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wh5tn" Oct 30 01:23:51.753227 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:51.753014 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a9c11a6-3a7f-473b-92c5-80a93a69bc34-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wh5tn\" (UID: \"3a9c11a6-3a7f-473b-92c5-80a93a69bc34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wh5tn" Oct 30 01:23:51.753227 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:51.753038 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3a9c11a6-3a7f-473b-92c5-80a93a69bc34-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wh5tn\" (UID: \"3a9c11a6-3a7f-473b-92c5-80a93a69bc34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wh5tn" Oct 30 01:23:51.753227 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:51.753060 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a9c11a6-3a7f-473b-92c5-80a93a69bc34-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wh5tn\" (UID: \"3a9c11a6-3a7f-473b-92c5-80a93a69bc34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wh5tn" Oct 30 01:23:51.753788 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:51.753749 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3a9c11a6-3a7f-473b-92c5-80a93a69bc34-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wh5tn\" (UID: \"3a9c11a6-3a7f-473b-92c5-80a93a69bc34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wh5tn" Oct 30 01:23:51.755245 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:51.755219 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a9c11a6-3a7f-473b-92c5-80a93a69bc34-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wh5tn\" (UID: \"3a9c11a6-3a7f-473b-92c5-80a93a69bc34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wh5tn" Oct 30 01:23:51.755400 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:51.755375 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3a9c11a6-3a7f-473b-92c5-80a93a69bc34-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wh5tn\" (UID: \"3a9c11a6-3a7f-473b-92c5-80a93a69bc34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wh5tn" Oct 30 01:23:51.755690 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:51.755670 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3a9c11a6-3a7f-473b-92c5-80a93a69bc34-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wh5tn\" (UID: \"3a9c11a6-3a7f-473b-92c5-80a93a69bc34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wh5tn" Oct 30 01:23:51.755773 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:51.755748 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3a9c11a6-3a7f-473b-92c5-80a93a69bc34-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wh5tn\" (UID: \"3a9c11a6-3a7f-473b-92c5-80a93a69bc34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wh5tn" Oct 30 01:23:51.774818 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:51.774790 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a9c11a6-3a7f-473b-92c5-80a93a69bc34-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wh5tn\" (UID: \"3a9c11a6-3a7f-473b-92c5-80a93a69bc34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wh5tn" Oct 30 01:23:51.776549 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:51.776511 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2xxq\" (UniqueName: \"kubernetes.io/projected/3a9c11a6-3a7f-473b-92c5-80a93a69bc34-kube-api-access-w2xxq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wh5tn\" (UID: \"3a9c11a6-3a7f-473b-92c5-80a93a69bc34\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wh5tn" Oct 30 01:23:51.947345 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:51.947315 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wh5tn" Oct 30 01:23:52.274936 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:52.274896 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wh5tn"] Oct 30 01:23:52.279320 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:23:52.279294 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a9c11a6_3a7f_473b_92c5_80a93a69bc34.slice/crio-e66d1f21ea4e397df49fee10e47a3dcab98fda7fa9f9fb6e36cd4f79cbd6247e WatchSource:0}: Error finding container e66d1f21ea4e397df49fee10e47a3dcab98fda7fa9f9fb6e36cd4f79cbd6247e: Status 404 returned error can't find the container with id e66d1f21ea4e397df49fee10e47a3dcab98fda7fa9f9fb6e36cd4f79cbd6247e Oct 30 01:23:52.281313 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:52.281280 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:23:52.281415 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:52.281363 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:23:52.825898 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:52.825860 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wh5tn" event={"ID":"3a9c11a6-3a7f-473b-92c5-80a93a69bc34","Type":"ContainerStarted","Data":"a87f5d94c19e00774fe96fec8ad35c20a8dfefc0a489d3410f3a92ea8b232a43"} Oct 30 01:23:52.826295 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:52.825933 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wh5tn" event={"ID":"3a9c11a6-3a7f-473b-92c5-80a93a69bc34","Type":"ContainerStarted","Data":"e66d1f21ea4e397df49fee10e47a3dcab98fda7fa9f9fb6e36cd4f79cbd6247e"} Oct 30 01:23:52.826295 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:52.825953 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wh5tn" Oct 30 01:23:53.832725 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:53.832689 6221 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-wh5tn container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:23:53.833182 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:53.832752 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wh5tn" podUID="3a9c11a6-3a7f-473b-92c5-80a93a69bc34" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:23:56.831977 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:56.831945 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wh5tn" Oct 30 01:23:56.899166 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:56.899114 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wh5tn" podStartSLOduration=5.899101055 podStartE2EDuration="5.899101055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:23:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:23:52.879051761 +0000 UTC m=+356.457339097" watchObservedRunningTime="2026-10-30 01:23:56.899101055 +0000 UTC m=+360.477388385" Oct 30 01:23:56.915260 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:56.915233 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-v26wx"] Oct 30 01:23:56.915432 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:56.915410 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v26wx" podUID="1665b186-6b0c-4cf4-8505-261e21f56cbf" containerName="discovery" containerID="cri-o://90c754399db642fd82a252e189a536e826ec5fbda048c03e11dd80fd9b5f0b13" gracePeriod=30 Oct 30 01:23:57.156026 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:57.156003 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v26wx" Oct 30 01:23:57.199379 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:57.199350 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1665b186-6b0c-4cf4-8505-261e21f56cbf-kube-api-access-snkln\" (UniqueName: \"kubernetes.io/projected/1665b186-6b0c-4cf4-8505-261e21f56cbf-kube-api-access-snkln\") pod \"1665b186-6b0c-4cf4-8505-261e21f56cbf\" (UID: \"1665b186-6b0c-4cf4-8505-261e21f56cbf\") " Oct 30 01:23:57.199557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:57.199408 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1665b186-6b0c-4cf4-8505-261e21f56cbf-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1665b186-6b0c-4cf4-8505-261e21f56cbf-local-certs\") pod \"1665b186-6b0c-4cf4-8505-261e21f56cbf\" (UID: \"1665b186-6b0c-4cf4-8505-261e21f56cbf\") " Oct 30 01:23:57.199557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:57.199439 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1665b186-6b0c-4cf4-8505-261e21f56cbf-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1665b186-6b0c-4cf4-8505-261e21f56cbf-istio-csr-dns-cert\") pod \"1665b186-6b0c-4cf4-8505-261e21f56cbf\" (UID: \"1665b186-6b0c-4cf4-8505-261e21f56cbf\") " Oct 30 01:23:57.199557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:57.199523 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1665b186-6b0c-4cf4-8505-261e21f56cbf-cacerts\" (UniqueName: \"kubernetes.io/secret/1665b186-6b0c-4cf4-8505-261e21f56cbf-cacerts\") pod \"1665b186-6b0c-4cf4-8505-261e21f56cbf\" (UID: \"1665b186-6b0c-4cf4-8505-261e21f56cbf\") " Oct 30 01:23:57.199733 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:57.199638 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1665b186-6b0c-4cf4-8505-261e21f56cbf-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1665b186-6b0c-4cf4-8505-261e21f56cbf-istio-csr-ca-configmap\") pod \"1665b186-6b0c-4cf4-8505-261e21f56cbf\" (UID: \"1665b186-6b0c-4cf4-8505-261e21f56cbf\") " Oct 30 01:23:57.199733 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:57.199681 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1665b186-6b0c-4cf4-8505-261e21f56cbf-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1665b186-6b0c-4cf4-8505-261e21f56cbf-istio-kubeconfig\") pod \"1665b186-6b0c-4cf4-8505-261e21f56cbf\" (UID: \"1665b186-6b0c-4cf4-8505-261e21f56cbf\") " Oct 30 01:23:57.199838 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:57.199768 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1665b186-6b0c-4cf4-8505-261e21f56cbf-istio-token\" (UniqueName: \"kubernetes.io/projected/1665b186-6b0c-4cf4-8505-261e21f56cbf-istio-token\") pod \"1665b186-6b0c-4cf4-8505-261e21f56cbf\" (UID: \"1665b186-6b0c-4cf4-8505-261e21f56cbf\") " Oct 30 01:23:57.200159 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:57.200128 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1665b186-6b0c-4cf4-8505-261e21f56cbf-istio-csr-ca-configmap" pod "1665b186-6b0c-4cf4-8505-261e21f56cbf" (UID: "1665b186-6b0c-4cf4-8505-261e21f56cbf"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:23:57.202725 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:57.202698 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1665b186-6b0c-4cf4-8505-261e21f56cbf-istio-csr-dns-cert" pod "1665b186-6b0c-4cf4-8505-261e21f56cbf" (UID: "1665b186-6b0c-4cf4-8505-261e21f56cbf"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:23:57.202833 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:57.202793 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1665b186-6b0c-4cf4-8505-261e21f56cbf-cacerts" pod "1665b186-6b0c-4cf4-8505-261e21f56cbf" (UID: "1665b186-6b0c-4cf4-8505-261e21f56cbf"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:23:57.202974 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:57.202950 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1665b186-6b0c-4cf4-8505-261e21f56cbf-kube-api-access-snkln" pod "1665b186-6b0c-4cf4-8505-261e21f56cbf" (UID: "1665b186-6b0c-4cf4-8505-261e21f56cbf"). InnerVolumeSpecName "kube-api-access-snkln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:57.203457 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:57.203428 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1665b186-6b0c-4cf4-8505-261e21f56cbf-istio-kubeconfig" pod "1665b186-6b0c-4cf4-8505-261e21f56cbf" (UID: "1665b186-6b0c-4cf4-8505-261e21f56cbf"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:23:57.203708 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:57.203688 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1665b186-6b0c-4cf4-8505-261e21f56cbf-local-certs" pod "1665b186-6b0c-4cf4-8505-261e21f56cbf" (UID: "1665b186-6b0c-4cf4-8505-261e21f56cbf"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:23:57.203811 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:57.203708 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1665b186-6b0c-4cf4-8505-261e21f56cbf-istio-token" pod "1665b186-6b0c-4cf4-8505-261e21f56cbf" (UID: "1665b186-6b0c-4cf4-8505-261e21f56cbf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:23:57.300553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:57.300499 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1665b186-6b0c-4cf4-8505-261e21f56cbf-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:23:57.300698 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:57.300558 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snkln\" (UniqueName: \"kubernetes.io/projected/1665b186-6b0c-4cf4-8505-261e21f56cbf-kube-api-access-snkln\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:23:57.300698 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:57.300575 6221 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1665b186-6b0c-4cf4-8505-261e21f56cbf-local-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:23:57.300698 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:57.300588 6221 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1665b186-6b0c-4cf4-8505-261e21f56cbf-istio-csr-dns-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:23:57.300698 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:57.300600 6221 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1665b186-6b0c-4cf4-8505-261e21f56cbf-cacerts\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:23:57.300698 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:57.300613 6221 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1665b186-6b0c-4cf4-8505-261e21f56cbf-istio-csr-ca-configmap\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:23:57.300698 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:57.300626 6221 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1665b186-6b0c-4cf4-8505-261e21f56cbf-istio-kubeconfig\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:23:57.848038 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:57.848002 6221 generic.go:356] "Generic (PLEG): container finished" podID="1665b186-6b0c-4cf4-8505-261e21f56cbf" containerID="90c754399db642fd82a252e189a536e826ec5fbda048c03e11dd80fd9b5f0b13" exitCode=0 Oct 30 01:23:57.848440 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:57.848063 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v26wx" Oct 30 01:23:57.848440 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:57.848092 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v26wx" event={"ID":"1665b186-6b0c-4cf4-8505-261e21f56cbf","Type":"ContainerDied","Data":"90c754399db642fd82a252e189a536e826ec5fbda048c03e11dd80fd9b5f0b13"} Oct 30 01:23:57.848440 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:57.848165 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-v26wx" event={"ID":"1665b186-6b0c-4cf4-8505-261e21f56cbf","Type":"ContainerDied","Data":"dc93afaed1bd39230835fb352996abdb3e891b33e124de860ed264d60084c257"} Oct 30 01:23:57.848440 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:57.848186 6221 scope.go:122] "RemoveContainer" containerID="90c754399db642fd82a252e189a536e826ec5fbda048c03e11dd80fd9b5f0b13" Oct 30 01:23:57.856365 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:57.856348 6221 scope.go:122] "RemoveContainer" containerID="90c754399db642fd82a252e189a536e826ec5fbda048c03e11dd80fd9b5f0b13" Oct 30 01:23:57.856653 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:23:57.856633 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90c754399db642fd82a252e189a536e826ec5fbda048c03e11dd80fd9b5f0b13\": container with ID starting with 90c754399db642fd82a252e189a536e826ec5fbda048c03e11dd80fd9b5f0b13 not found: ID does not exist" containerID="90c754399db642fd82a252e189a536e826ec5fbda048c03e11dd80fd9b5f0b13" Oct 30 01:23:57.856702 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:57.856661 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90c754399db642fd82a252e189a536e826ec5fbda048c03e11dd80fd9b5f0b13"} err="failed to get container status \"90c754399db642fd82a252e189a536e826ec5fbda048c03e11dd80fd9b5f0b13\": rpc error: code = NotFound desc = could not find container \"90c754399db642fd82a252e189a536e826ec5fbda048c03e11dd80fd9b5f0b13\": container with ID starting with 90c754399db642fd82a252e189a536e826ec5fbda048c03e11dd80fd9b5f0b13 not found: ID does not exist" Oct 30 01:23:57.908717 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:57.908685 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-v26wx"] Oct 30 01:23:57.912896 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:57.912874 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-v26wx"] Oct 30 01:23:58.544206 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:23:58.544169 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1665b186-6b0c-4cf4-8505-261e21f56cbf" path="/var/lib/kubelet/pods/1665b186-6b0c-4cf4-8505-261e21f56cbf/volumes" Oct 30 01:24:20.010909 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:20.010823 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-849f8d77f4-7p485"] Oct 30 01:24:20.011412 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:20.011132 6221 cpu_manager.go:414] "containerMap: removing container" podUID="1665b186-6b0c-4cf4-8505-261e21f56cbf" containerName="discovery" Oct 30 01:24:20.011412 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:20.011144 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1665b186-6b0c-4cf4-8505-261e21f56cbf" containerName="discovery" Oct 30 01:24:20.011412 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:20.011214 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="1665b186-6b0c-4cf4-8505-261e21f56cbf" containerName="discovery" Oct 30 01:24:20.014141 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:20.014125 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-849f8d77f4-7p485" Oct 30 01:24:20.029515 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:20.029494 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 01:24:20.032027 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:20.032006 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 01:24:20.043689 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:20.043665 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-849f8d77f4-7p485"] Oct 30 01:24:20.044693 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:20.044661 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-x5cst\"" Oct 30 01:24:20.044803 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:20.044722 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 01:24:20.044803 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:20.044729 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 01:24:20.093859 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:20.093820 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zwq6\" (UniqueName: \"kubernetes.io/projected/e1ce462a-e3ae-4fde-87bf-d33af3510e7c-kube-api-access-8zwq6\") pod \"maas-api-849f8d77f4-7p485\" (UID: \"e1ce462a-e3ae-4fde-87bf-d33af3510e7c\") " pod="odh-ai-gateway-infra/maas-api-849f8d77f4-7p485" Oct 30 01:24:20.094033 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:20.093876 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1ce462a-e3ae-4fde-87bf-d33af3510e7c-openshift-service-ca\") pod \"maas-api-849f8d77f4-7p485\" (UID: \"e1ce462a-e3ae-4fde-87bf-d33af3510e7c\") " pod="odh-ai-gateway-infra/maas-api-849f8d77f4-7p485" Oct 30 01:24:20.094033 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:20.093954 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1ce462a-e3ae-4fde-87bf-d33af3510e7c-metrics-certs\") pod \"maas-api-849f8d77f4-7p485\" (UID: \"e1ce462a-e3ae-4fde-87bf-d33af3510e7c\") " pod="odh-ai-gateway-infra/maas-api-849f8d77f4-7p485" Oct 30 01:24:20.094033 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:20.093993 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1ce462a-e3ae-4fde-87bf-d33af3510e7c-maas-api-tls\") pod \"maas-api-849f8d77f4-7p485\" (UID: \"e1ce462a-e3ae-4fde-87bf-d33af3510e7c\") " pod="odh-ai-gateway-infra/maas-api-849f8d77f4-7p485" Oct 30 01:24:20.194722 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:20.194688 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1ce462a-e3ae-4fde-87bf-d33af3510e7c-maas-api-tls\") pod \"maas-api-849f8d77f4-7p485\" (UID: \"e1ce462a-e3ae-4fde-87bf-d33af3510e7c\") " pod="odh-ai-gateway-infra/maas-api-849f8d77f4-7p485" Oct 30 01:24:20.194908 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:20.194742 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zwq6\" (UniqueName: \"kubernetes.io/projected/e1ce462a-e3ae-4fde-87bf-d33af3510e7c-kube-api-access-8zwq6\") pod \"maas-api-849f8d77f4-7p485\" (UID: \"e1ce462a-e3ae-4fde-87bf-d33af3510e7c\") " pod="odh-ai-gateway-infra/maas-api-849f8d77f4-7p485" Oct 30 01:24:20.194908 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:20.194784 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1ce462a-e3ae-4fde-87bf-d33af3510e7c-openshift-service-ca\") pod \"maas-api-849f8d77f4-7p485\" (UID: \"e1ce462a-e3ae-4fde-87bf-d33af3510e7c\") " pod="odh-ai-gateway-infra/maas-api-849f8d77f4-7p485" Oct 30 01:24:20.194908 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:20.194806 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1ce462a-e3ae-4fde-87bf-d33af3510e7c-metrics-certs\") pod \"maas-api-849f8d77f4-7p485\" (UID: \"e1ce462a-e3ae-4fde-87bf-d33af3510e7c\") " pod="odh-ai-gateway-infra/maas-api-849f8d77f4-7p485" Oct 30 01:24:20.195413 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:20.195391 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1ce462a-e3ae-4fde-87bf-d33af3510e7c-openshift-service-ca\") pod \"maas-api-849f8d77f4-7p485\" (UID: \"e1ce462a-e3ae-4fde-87bf-d33af3510e7c\") " pod="odh-ai-gateway-infra/maas-api-849f8d77f4-7p485" Oct 30 01:24:20.197248 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:20.197219 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1ce462a-e3ae-4fde-87bf-d33af3510e7c-metrics-certs\") pod \"maas-api-849f8d77f4-7p485\" (UID: \"e1ce462a-e3ae-4fde-87bf-d33af3510e7c\") " pod="odh-ai-gateway-infra/maas-api-849f8d77f4-7p485" Oct 30 01:24:20.197383 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:20.197360 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1ce462a-e3ae-4fde-87bf-d33af3510e7c-maas-api-tls\") pod \"maas-api-849f8d77f4-7p485\" (UID: \"e1ce462a-e3ae-4fde-87bf-d33af3510e7c\") " pod="odh-ai-gateway-infra/maas-api-849f8d77f4-7p485" Oct 30 01:24:20.214036 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:20.214012 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zwq6\" (UniqueName: \"kubernetes.io/projected/e1ce462a-e3ae-4fde-87bf-d33af3510e7c-kube-api-access-8zwq6\") pod \"maas-api-849f8d77f4-7p485\" (UID: \"e1ce462a-e3ae-4fde-87bf-d33af3510e7c\") " pod="odh-ai-gateway-infra/maas-api-849f8d77f4-7p485" Oct 30 01:24:20.327407 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:20.327374 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-849f8d77f4-7p485" Oct 30 01:24:20.624505 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:20.624420 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-849f8d77f4-7p485"] Oct 30 01:24:20.628575 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:24:20.628548 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1ce462a_e3ae_4fde_87bf_d33af3510e7c.slice/crio-8919010ac49bac0b769516f9100626b55708efcd17d2f45d30282019d73974fa WatchSource:0}: Error finding container 8919010ac49bac0b769516f9100626b55708efcd17d2f45d30282019d73974fa: Status 404 returned error can't find the container with id 8919010ac49bac0b769516f9100626b55708efcd17d2f45d30282019d73974fa Oct 30 01:24:20.939789 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:20.939702 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-849f8d77f4-7p485" event={"ID":"e1ce462a-e3ae-4fde-87bf-d33af3510e7c","Type":"ContainerStarted","Data":"8919010ac49bac0b769516f9100626b55708efcd17d2f45d30282019d73974fa"} Oct 30 01:24:21.501248 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:24:21.501208 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:24:21.742656 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:21.742624 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:24:22.949158 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:22.949119 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-849f8d77f4-7p485" event={"ID":"e1ce462a-e3ae-4fde-87bf-d33af3510e7c","Type":"ContainerStarted","Data":"9b00e0a75be661a517d6e765d9309185131697231a6d0e76c6e7fc174c59079f"} Oct 30 01:24:22.949541 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:22.949244 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-849f8d77f4-7p485" Oct 30 01:24:22.988320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:22.988275 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-849f8d77f4-7p485" podStartSLOduration=1.773372852 podStartE2EDuration="3.988262129s" totalImagesPullingTime="2.214889277s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:24:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:24:22.987684139 +0000 UTC m=+386.565971474" watchObservedRunningTime="2026-10-30 01:24:22.988262129 +0000 UTC m=+386.566549462" Oct 30 01:24:28.959178 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:28.959145 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-849f8d77f4-7p485" Oct 30 01:24:29.877546 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:29.877493 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7b8b8d5c6c-dm9dk"] Oct 30 01:24:29.880903 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:29.880880 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b8b8d5c6c-dm9dk" Oct 30 01:24:29.911264 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:29.911159 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b8b8d5c6c-dm9dk"] Oct 30 01:24:29.983104 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:29.983072 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3236f15b-ffde-4a60-ba93-fb48f9befbb0-metrics-certs\") pod \"maas-api-7b8b8d5c6c-dm9dk\" (UID: \"3236f15b-ffde-4a60-ba93-fb48f9befbb0\") " pod="odh-ai-gateway-infra/maas-api-7b8b8d5c6c-dm9dk" Oct 30 01:24:29.983520 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:29.983114 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcqrd\" (UniqueName: \"kubernetes.io/projected/3236f15b-ffde-4a60-ba93-fb48f9befbb0-kube-api-access-jcqrd\") pod \"maas-api-7b8b8d5c6c-dm9dk\" (UID: \"3236f15b-ffde-4a60-ba93-fb48f9befbb0\") " pod="odh-ai-gateway-infra/maas-api-7b8b8d5c6c-dm9dk" Oct 30 01:24:29.983520 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:29.983232 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3236f15b-ffde-4a60-ba93-fb48f9befbb0-openshift-service-ca\") pod \"maas-api-7b8b8d5c6c-dm9dk\" (UID: \"3236f15b-ffde-4a60-ba93-fb48f9befbb0\") " pod="odh-ai-gateway-infra/maas-api-7b8b8d5c6c-dm9dk" Oct 30 01:24:29.983520 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:29.983261 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3236f15b-ffde-4a60-ba93-fb48f9befbb0-maas-api-tls\") pod \"maas-api-7b8b8d5c6c-dm9dk\" (UID: \"3236f15b-ffde-4a60-ba93-fb48f9befbb0\") " pod="odh-ai-gateway-infra/maas-api-7b8b8d5c6c-dm9dk" Oct 30 01:24:30.084275 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:30.084229 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3236f15b-ffde-4a60-ba93-fb48f9befbb0-metrics-certs\") pod \"maas-api-7b8b8d5c6c-dm9dk\" (UID: \"3236f15b-ffde-4a60-ba93-fb48f9befbb0\") " pod="odh-ai-gateway-infra/maas-api-7b8b8d5c6c-dm9dk" Oct 30 01:24:30.084456 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:30.084293 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcqrd\" (UniqueName: \"kubernetes.io/projected/3236f15b-ffde-4a60-ba93-fb48f9befbb0-kube-api-access-jcqrd\") pod \"maas-api-7b8b8d5c6c-dm9dk\" (UID: \"3236f15b-ffde-4a60-ba93-fb48f9befbb0\") " pod="odh-ai-gateway-infra/maas-api-7b8b8d5c6c-dm9dk" Oct 30 01:24:30.084456 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:30.084376 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3236f15b-ffde-4a60-ba93-fb48f9befbb0-openshift-service-ca\") pod \"maas-api-7b8b8d5c6c-dm9dk\" (UID: \"3236f15b-ffde-4a60-ba93-fb48f9befbb0\") " pod="odh-ai-gateway-infra/maas-api-7b8b8d5c6c-dm9dk" Oct 30 01:24:30.084456 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:30.084411 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3236f15b-ffde-4a60-ba93-fb48f9befbb0-maas-api-tls\") pod \"maas-api-7b8b8d5c6c-dm9dk\" (UID: \"3236f15b-ffde-4a60-ba93-fb48f9befbb0\") " pod="odh-ai-gateway-infra/maas-api-7b8b8d5c6c-dm9dk" Oct 30 01:24:30.085055 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:30.085031 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3236f15b-ffde-4a60-ba93-fb48f9befbb0-openshift-service-ca\") pod \"maas-api-7b8b8d5c6c-dm9dk\" (UID: \"3236f15b-ffde-4a60-ba93-fb48f9befbb0\") " pod="odh-ai-gateway-infra/maas-api-7b8b8d5c6c-dm9dk" Oct 30 01:24:30.086943 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:30.086919 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3236f15b-ffde-4a60-ba93-fb48f9befbb0-maas-api-tls\") pod \"maas-api-7b8b8d5c6c-dm9dk\" (UID: \"3236f15b-ffde-4a60-ba93-fb48f9befbb0\") " pod="odh-ai-gateway-infra/maas-api-7b8b8d5c6c-dm9dk" Oct 30 01:24:30.087044 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:30.086961 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3236f15b-ffde-4a60-ba93-fb48f9befbb0-metrics-certs\") pod \"maas-api-7b8b8d5c6c-dm9dk\" (UID: \"3236f15b-ffde-4a60-ba93-fb48f9befbb0\") " pod="odh-ai-gateway-infra/maas-api-7b8b8d5c6c-dm9dk" Oct 30 01:24:30.103635 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:30.103600 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcqrd\" (UniqueName: \"kubernetes.io/projected/3236f15b-ffde-4a60-ba93-fb48f9befbb0-kube-api-access-jcqrd\") pod \"maas-api-7b8b8d5c6c-dm9dk\" (UID: \"3236f15b-ffde-4a60-ba93-fb48f9befbb0\") " pod="odh-ai-gateway-infra/maas-api-7b8b8d5c6c-dm9dk" Oct 30 01:24:30.192822 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:30.192728 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7b8b8d5c6c-dm9dk" Oct 30 01:24:30.490992 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:30.490889 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7b8b8d5c6c-dm9dk"] Oct 30 01:24:30.495385 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:24:30.495351 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3236f15b_ffde_4a60_ba93_fb48f9befbb0.slice/crio-484557c8b957b38e16955d807c5bb89febe7ef3202903847519c9147e5ed454f WatchSource:0}: Error finding container 484557c8b957b38e16955d807c5bb89febe7ef3202903847519c9147e5ed454f: Status 404 returned error can't find the container with id 484557c8b957b38e16955d807c5bb89febe7ef3202903847519c9147e5ed454f Oct 30 01:24:30.980834 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:30.980796 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b8b8d5c6c-dm9dk" event={"ID":"3236f15b-ffde-4a60-ba93-fb48f9befbb0","Type":"ContainerStarted","Data":"6c475843c075c590874622c1afbf96dedd903e76f367180206e706d004833374"} Oct 30 01:24:30.981030 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:30.980849 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7b8b8d5c6c-dm9dk" event={"ID":"3236f15b-ffde-4a60-ba93-fb48f9befbb0","Type":"ContainerStarted","Data":"484557c8b957b38e16955d807c5bb89febe7ef3202903847519c9147e5ed454f"} Oct 30 01:24:30.981030 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:30.980913 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7b8b8d5c6c-dm9dk" Oct 30 01:24:31.026975 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:31.026923 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7b8b8d5c6c-dm9dk" podStartSLOduration=1.6425281269999998 podStartE2EDuration="2.026909108s" totalImagesPullingTime="384.380981ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:24:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:24:31.026457388 +0000 UTC m=+394.604744727" watchObservedRunningTime="2026-10-30 01:24:31.026909108 +0000 UTC m=+394.605196444" Oct 30 01:24:36.990401 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:36.990370 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7b8b8d5c6c-dm9dk" Oct 30 01:24:37.058606 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:37.058578 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-849f8d77f4-7p485"] Oct 30 01:24:37.058805 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:37.058765 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-849f8d77f4-7p485" podUID="e1ce462a-e3ae-4fde-87bf-d33af3510e7c" containerName="maas-api" containerID="cri-o://9b00e0a75be661a517d6e765d9309185131697231a6d0e76c6e7fc174c59079f" gracePeriod=30 Oct 30 01:24:37.295851 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:37.295828 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-849f8d77f4-7p485" Oct 30 01:24:37.346079 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:37.346052 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1ce462a-e3ae-4fde-87bf-d33af3510e7c-kube-api-access-8zwq6\" (UniqueName: \"kubernetes.io/projected/e1ce462a-e3ae-4fde-87bf-d33af3510e7c-kube-api-access-8zwq6\") pod \"e1ce462a-e3ae-4fde-87bf-d33af3510e7c\" (UID: \"e1ce462a-e3ae-4fde-87bf-d33af3510e7c\") " Oct 30 01:24:37.346252 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:37.346085 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1ce462a-e3ae-4fde-87bf-d33af3510e7c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1ce462a-e3ae-4fde-87bf-d33af3510e7c-metrics-certs\") pod \"e1ce462a-e3ae-4fde-87bf-d33af3510e7c\" (UID: \"e1ce462a-e3ae-4fde-87bf-d33af3510e7c\") " Oct 30 01:24:37.346252 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:37.346103 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1ce462a-e3ae-4fde-87bf-d33af3510e7c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1ce462a-e3ae-4fde-87bf-d33af3510e7c-maas-api-tls\") pod \"e1ce462a-e3ae-4fde-87bf-d33af3510e7c\" (UID: \"e1ce462a-e3ae-4fde-87bf-d33af3510e7c\") " Oct 30 01:24:37.346252 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:37.346178 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1ce462a-e3ae-4fde-87bf-d33af3510e7c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1ce462a-e3ae-4fde-87bf-d33af3510e7c-openshift-service-ca\") pod \"e1ce462a-e3ae-4fde-87bf-d33af3510e7c\" (UID: \"e1ce462a-e3ae-4fde-87bf-d33af3510e7c\") " Oct 30 01:24:37.346598 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:37.346571 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1ce462a-e3ae-4fde-87bf-d33af3510e7c-openshift-service-ca" pod "e1ce462a-e3ae-4fde-87bf-d33af3510e7c" (UID: "e1ce462a-e3ae-4fde-87bf-d33af3510e7c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:24:37.348399 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:37.348364 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1ce462a-e3ae-4fde-87bf-d33af3510e7c-kube-api-access-8zwq6" pod "e1ce462a-e3ae-4fde-87bf-d33af3510e7c" (UID: "e1ce462a-e3ae-4fde-87bf-d33af3510e7c"). InnerVolumeSpecName "kube-api-access-8zwq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:24:37.348399 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:37.348389 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1ce462a-e3ae-4fde-87bf-d33af3510e7c-metrics-certs" pod "e1ce462a-e3ae-4fde-87bf-d33af3510e7c" (UID: "e1ce462a-e3ae-4fde-87bf-d33af3510e7c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:24:37.348580 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:37.348449 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1ce462a-e3ae-4fde-87bf-d33af3510e7c-maas-api-tls" pod "e1ce462a-e3ae-4fde-87bf-d33af3510e7c" (UID: "e1ce462a-e3ae-4fde-87bf-d33af3510e7c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:24:37.446755 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:37.446703 6221 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1ce462a-e3ae-4fde-87bf-d33af3510e7c-openshift-service-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:24:37.446755 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:37.446748 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zwq6\" (UniqueName: \"kubernetes.io/projected/e1ce462a-e3ae-4fde-87bf-d33af3510e7c-kube-api-access-8zwq6\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:24:37.446755 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:37.446763 6221 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1ce462a-e3ae-4fde-87bf-d33af3510e7c-metrics-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:24:37.446983 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:37.446783 6221 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1ce462a-e3ae-4fde-87bf-d33af3510e7c-maas-api-tls\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:24:38.008944 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:38.008909 6221 generic.go:356] "Generic (PLEG): container finished" podID="e1ce462a-e3ae-4fde-87bf-d33af3510e7c" containerID="9b00e0a75be661a517d6e765d9309185131697231a6d0e76c6e7fc174c59079f" exitCode=0 Oct 30 01:24:38.009381 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:38.008969 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-849f8d77f4-7p485" event={"ID":"e1ce462a-e3ae-4fde-87bf-d33af3510e7c","Type":"ContainerDied","Data":"9b00e0a75be661a517d6e765d9309185131697231a6d0e76c6e7fc174c59079f"} Oct 30 01:24:38.009381 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:38.008982 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-849f8d77f4-7p485" Oct 30 01:24:38.009381 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:38.009035 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-849f8d77f4-7p485" event={"ID":"e1ce462a-e3ae-4fde-87bf-d33af3510e7c","Type":"ContainerDied","Data":"8919010ac49bac0b769516f9100626b55708efcd17d2f45d30282019d73974fa"} Oct 30 01:24:38.009381 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:38.009058 6221 scope.go:122] "RemoveContainer" containerID="9b00e0a75be661a517d6e765d9309185131697231a6d0e76c6e7fc174c59079f" Oct 30 01:24:38.017707 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:38.017688 6221 scope.go:122] "RemoveContainer" containerID="9b00e0a75be661a517d6e765d9309185131697231a6d0e76c6e7fc174c59079f" Oct 30 01:24:38.017990 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:24:38.017966 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b00e0a75be661a517d6e765d9309185131697231a6d0e76c6e7fc174c59079f\": container with ID starting with 9b00e0a75be661a517d6e765d9309185131697231a6d0e76c6e7fc174c59079f not found: ID does not exist" containerID="9b00e0a75be661a517d6e765d9309185131697231a6d0e76c6e7fc174c59079f" Oct 30 01:24:38.018043 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:38.017998 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b00e0a75be661a517d6e765d9309185131697231a6d0e76c6e7fc174c59079f"} err="failed to get container status \"9b00e0a75be661a517d6e765d9309185131697231a6d0e76c6e7fc174c59079f\": rpc error: code = NotFound desc = could not find container \"9b00e0a75be661a517d6e765d9309185131697231a6d0e76c6e7fc174c59079f\": container with ID starting with 9b00e0a75be661a517d6e765d9309185131697231a6d0e76c6e7fc174c59079f not found: ID does not exist" Oct 30 01:24:38.058107 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:38.058079 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-849f8d77f4-7p485"] Oct 30 01:24:38.070232 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:38.070208 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-849f8d77f4-7p485"] Oct 30 01:24:38.544614 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:24:38.544578 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1ce462a-e3ae-4fde-87bf-d33af3510e7c" path="/var/lib/kubelet/pods/e1ce462a-e3ae-4fde-87bf-d33af3510e7c/volumes" Oct 30 01:24:51.491938 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:24:51.491899 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:25:21.493457 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:25:21.493422 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:25:26.418986 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:26.418948 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-dn6pg"] Oct 30 01:25:26.419394 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:26.419281 6221 cpu_manager.go:414] "containerMap: removing container" podUID="e1ce462a-e3ae-4fde-87bf-d33af3510e7c" containerName="maas-api" Oct 30 01:25:26.419394 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:26.419297 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1ce462a-e3ae-4fde-87bf-d33af3510e7c" containerName="maas-api" Oct 30 01:25:26.419394 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:26.419379 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1ce462a-e3ae-4fde-87bf-d33af3510e7c" containerName="maas-api" Oct 30 01:25:26.421398 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:26.421375 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-dn6pg" Oct 30 01:25:26.452007 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:26.451961 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 01:25:26.452007 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:26.451961 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 01:25:26.452191 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:26.452041 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 01:25:26.452191 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:26.452167 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 01:25:26.452372 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:26.452355 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-w74lt\"" Oct 30 01:25:26.453511 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:26.453485 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-dn6pg"] Oct 30 01:25:26.549782 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:26.549754 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzdpv\" (UniqueName: \"kubernetes.io/projected/219e3edc-8211-4364-92ee-203bfb1e914e-kube-api-access-tzdpv\") pod \"ai-gateway-controller-7d85675f98-dn6pg\" (UID: \"219e3edc-8211-4364-92ee-203bfb1e914e\") " pod="opendatahub/ai-gateway-controller-7d85675f98-dn6pg" Oct 30 01:25:26.549949 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:26.549804 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/219e3edc-8211-4364-92ee-203bfb1e914e-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-dn6pg\" (UID: \"219e3edc-8211-4364-92ee-203bfb1e914e\") " pod="opendatahub/ai-gateway-controller-7d85675f98-dn6pg" Oct 30 01:25:26.650436 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:26.650397 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzdpv\" (UniqueName: \"kubernetes.io/projected/219e3edc-8211-4364-92ee-203bfb1e914e-kube-api-access-tzdpv\") pod \"ai-gateway-controller-7d85675f98-dn6pg\" (UID: \"219e3edc-8211-4364-92ee-203bfb1e914e\") " pod="opendatahub/ai-gateway-controller-7d85675f98-dn6pg" Oct 30 01:25:26.650698 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:26.650449 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/219e3edc-8211-4364-92ee-203bfb1e914e-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-dn6pg\" (UID: \"219e3edc-8211-4364-92ee-203bfb1e914e\") " pod="opendatahub/ai-gateway-controller-7d85675f98-dn6pg" Oct 30 01:25:26.652875 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:26.652854 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/219e3edc-8211-4364-92ee-203bfb1e914e-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-dn6pg\" (UID: \"219e3edc-8211-4364-92ee-203bfb1e914e\") " pod="opendatahub/ai-gateway-controller-7d85675f98-dn6pg" Oct 30 01:25:26.670132 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:26.670076 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzdpv\" (UniqueName: \"kubernetes.io/projected/219e3edc-8211-4364-92ee-203bfb1e914e-kube-api-access-tzdpv\") pod \"ai-gateway-controller-7d85675f98-dn6pg\" (UID: \"219e3edc-8211-4364-92ee-203bfb1e914e\") " pod="opendatahub/ai-gateway-controller-7d85675f98-dn6pg" Oct 30 01:25:26.732436 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:26.732402 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-dn6pg" Oct 30 01:25:27.025066 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:27.024996 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-dn6pg"] Oct 30 01:25:27.029170 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:25:27.029144 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod219e3edc_8211_4364_92ee_203bfb1e914e.slice/crio-21582b0ebf27545862e6ebd6e36e75e998d65ecec15bade6506a210c6b0ed293 WatchSource:0}: Error finding container 21582b0ebf27545862e6ebd6e36e75e998d65ecec15bade6506a210c6b0ed293: Status 404 returned error can't find the container with id 21582b0ebf27545862e6ebd6e36e75e998d65ecec15bade6506a210c6b0ed293 Oct 30 01:25:27.203826 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:27.203793 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-dn6pg" event={"ID":"219e3edc-8211-4364-92ee-203bfb1e914e","Type":"ContainerStarted","Data":"21582b0ebf27545862e6ebd6e36e75e998d65ecec15bade6506a210c6b0ed293"} Oct 30 01:25:29.212804 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:29.212763 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-dn6pg" event={"ID":"219e3edc-8211-4364-92ee-203bfb1e914e","Type":"ContainerStarted","Data":"740f9023f4fda8df6904e3f71f7a73f6d3c15a3fe27482f8db774437135733af"} Oct 30 01:25:29.213220 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:29.213024 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-dn6pg" Oct 30 01:25:29.250847 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:29.250799 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-dn6pg" podStartSLOduration=1.801093981 podStartE2EDuration="3.250787811s" totalImagesPullingTime="1.44969383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:25:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:25:29.250521257 +0000 UTC m=+452.828808605" watchObservedRunningTime="2026-10-30 01:25:29.250787811 +0000 UTC m=+452.829075147" Oct 30 01:25:29.716103 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:29.716068 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-wp6mf"] Oct 30 01:25:29.718494 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:29.718475 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-wp6mf" Oct 30 01:25:29.735774 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:29.735740 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-p5nqr\"" Oct 30 01:25:29.735904 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:29.735824 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 01:25:29.748991 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:29.748958 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-wp6mf"] Oct 30 01:25:29.774841 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:29.774810 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7qgc\" (UniqueName: \"kubernetes.io/projected/fc46a03a-6581-4d32-8ed7-a4d5c53625b4-kube-api-access-p7qgc\") pod \"payload-processing-5f78687d5b-wp6mf\" (UID: \"fc46a03a-6581-4d32-8ed7-a4d5c53625b4\") " pod="openshift-ingress/payload-processing-5f78687d5b-wp6mf" Oct 30 01:25:29.774996 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:29.774855 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc46a03a-6581-4d32-8ed7-a4d5c53625b4-plugins-config-volume\") pod \"payload-processing-5f78687d5b-wp6mf\" (UID: \"fc46a03a-6581-4d32-8ed7-a4d5c53625b4\") " pod="openshift-ingress/payload-processing-5f78687d5b-wp6mf" Oct 30 01:25:29.777579 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:29.777554 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-728tk"] Oct 30 01:25:29.779718 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:29.779702 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-728tk" Oct 30 01:25:29.798520 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:29.798494 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-728tk"] Oct 30 01:25:29.876111 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:29.876068 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7qgc\" (UniqueName: \"kubernetes.io/projected/fc46a03a-6581-4d32-8ed7-a4d5c53625b4-kube-api-access-p7qgc\") pod \"payload-processing-5f78687d5b-wp6mf\" (UID: \"fc46a03a-6581-4d32-8ed7-a4d5c53625b4\") " pod="openshift-ingress/payload-processing-5f78687d5b-wp6mf" Oct 30 01:25:29.876273 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:29.876130 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc46a03a-6581-4d32-8ed7-a4d5c53625b4-plugins-config-volume\") pod \"payload-processing-5f78687d5b-wp6mf\" (UID: \"fc46a03a-6581-4d32-8ed7-a4d5c53625b4\") " pod="openshift-ingress/payload-processing-5f78687d5b-wp6mf" Oct 30 01:25:29.876273 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:29.876175 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c0844ce-ae06-46a1-8819-decfc87c21e0-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-728tk\" (UID: \"2c0844ce-ae06-46a1-8819-decfc87c21e0\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-728tk" Oct 30 01:25:29.876273 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:29.876192 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhtcj\" (UniqueName: \"kubernetes.io/projected/2c0844ce-ae06-46a1-8819-decfc87c21e0-kube-api-access-xhtcj\") pod \"payload-pre-processing-7b98bb678c-728tk\" (UID: \"2c0844ce-ae06-46a1-8819-decfc87c21e0\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-728tk" Oct 30 01:25:29.876889 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:29.876870 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc46a03a-6581-4d32-8ed7-a4d5c53625b4-plugins-config-volume\") pod \"payload-processing-5f78687d5b-wp6mf\" (UID: \"fc46a03a-6581-4d32-8ed7-a4d5c53625b4\") " pod="openshift-ingress/payload-processing-5f78687d5b-wp6mf" Oct 30 01:25:29.894867 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:29.894840 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7qgc\" (UniqueName: \"kubernetes.io/projected/fc46a03a-6581-4d32-8ed7-a4d5c53625b4-kube-api-access-p7qgc\") pod \"payload-processing-5f78687d5b-wp6mf\" (UID: \"fc46a03a-6581-4d32-8ed7-a4d5c53625b4\") " pod="openshift-ingress/payload-processing-5f78687d5b-wp6mf" Oct 30 01:25:29.976880 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:29.976802 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c0844ce-ae06-46a1-8819-decfc87c21e0-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-728tk\" (UID: \"2c0844ce-ae06-46a1-8819-decfc87c21e0\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-728tk" Oct 30 01:25:29.976880 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:29.976836 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhtcj\" (UniqueName: \"kubernetes.io/projected/2c0844ce-ae06-46a1-8819-decfc87c21e0-kube-api-access-xhtcj\") pod \"payload-pre-processing-7b98bb678c-728tk\" (UID: \"2c0844ce-ae06-46a1-8819-decfc87c21e0\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-728tk" Oct 30 01:25:29.977473 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:29.977453 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c0844ce-ae06-46a1-8819-decfc87c21e0-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-728tk\" (UID: \"2c0844ce-ae06-46a1-8819-decfc87c21e0\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-728tk" Oct 30 01:25:30.002241 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:30.002208 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhtcj\" (UniqueName: \"kubernetes.io/projected/2c0844ce-ae06-46a1-8819-decfc87c21e0-kube-api-access-xhtcj\") pod \"payload-pre-processing-7b98bb678c-728tk\" (UID: \"2c0844ce-ae06-46a1-8819-decfc87c21e0\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-728tk" Oct 30 01:25:30.030098 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:30.030063 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-wp6mf" Oct 30 01:25:30.093260 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:30.090840 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-728tk" Oct 30 01:25:30.324364 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:30.324332 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-wp6mf"] Oct 30 01:25:30.328457 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:25:30.328430 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc46a03a_6581_4d32_8ed7_a4d5c53625b4.slice/crio-9fbebd6bfdf3398b96e0004b8ec75b68392077797713a2656e58e78ee19c35f5 WatchSource:0}: Error finding container 9fbebd6bfdf3398b96e0004b8ec75b68392077797713a2656e58e78ee19c35f5: Status 404 returned error can't find the container with id 9fbebd6bfdf3398b96e0004b8ec75b68392077797713a2656e58e78ee19c35f5 Oct 30 01:25:30.391511 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:30.391481 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-728tk"] Oct 30 01:25:30.396749 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:25:30.396717 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c0844ce_ae06_46a1_8819_decfc87c21e0.slice/crio-2b70e8671dfeff921347bddb9abbb8c6fd44c8abc27a915f17bcfbb3fd7f811b WatchSource:0}: Error finding container 2b70e8671dfeff921347bddb9abbb8c6fd44c8abc27a915f17bcfbb3fd7f811b: Status 404 returned error can't find the container with id 2b70e8671dfeff921347bddb9abbb8c6fd44c8abc27a915f17bcfbb3fd7f811b Oct 30 01:25:31.224012 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:31.223965 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-728tk" event={"ID":"2c0844ce-ae06-46a1-8819-decfc87c21e0","Type":"ContainerStarted","Data":"2b70e8671dfeff921347bddb9abbb8c6fd44c8abc27a915f17bcfbb3fd7f811b"} Oct 30 01:25:31.225364 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:31.225331 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-wp6mf" event={"ID":"fc46a03a-6581-4d32-8ed7-a4d5c53625b4","Type":"ContainerStarted","Data":"9fbebd6bfdf3398b96e0004b8ec75b68392077797713a2656e58e78ee19c35f5"} Oct 30 01:25:33.235941 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:33.235890 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-wp6mf" event={"ID":"fc46a03a-6581-4d32-8ed7-a4d5c53625b4","Type":"ContainerStarted","Data":"a097877df0337eb5c1d87d53dea44b044c235530139d7474d769bc0aaf5cfbe1"} Oct 30 01:25:33.236417 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:33.235966 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-wp6mf" Oct 30 01:25:33.237356 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:33.237338 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-728tk" event={"ID":"2c0844ce-ae06-46a1-8819-decfc87c21e0","Type":"ContainerStarted","Data":"8caa1985f507dfd5384236f54ba65a7a2e7536b31d030cb2bb5d4c3489a10c2c"} Oct 30 01:25:33.237460 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:33.237450 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-728tk" Oct 30 01:25:33.273819 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:33.273777 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-wp6mf" podStartSLOduration=2.095092233 podStartE2EDuration="4.273765078s" totalImagesPullingTime="2.178672845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:25:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:25:33.272940148 +0000 UTC m=+456.851227462" watchObservedRunningTime="2026-10-30 01:25:33.273765078 +0000 UTC m=+456.852052414" Oct 30 01:25:33.310189 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:33.310150 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-728tk" podStartSLOduration=2.130543432 podStartE2EDuration="4.310138465s" totalImagesPullingTime="2.179595033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:25:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:25:33.309836575 +0000 UTC m=+456.888123912" watchObservedRunningTime="2026-10-30 01:25:33.310138465 +0000 UTC m=+456.888425798" Oct 30 01:25:40.218496 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:40.218421 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-dn6pg" Oct 30 01:25:40.522422 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:40.522344 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7ddbcdf875-d8wmr"] Oct 30 01:25:40.525672 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:40.525651 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7ddbcdf875-d8wmr" Oct 30 01:25:40.554619 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:40.554589 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7ddbcdf875-d8wmr"] Oct 30 01:25:40.566752 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:40.566725 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daa56966-b092-4927-bac7-11fa18b39d00-metrics-certs\") pod \"ai-gateway-controller-7ddbcdf875-d8wmr\" (UID: \"daa56966-b092-4927-bac7-11fa18b39d00\") " pod="opendatahub/ai-gateway-controller-7ddbcdf875-d8wmr" Oct 30 01:25:40.566885 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:40.566757 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr7h4\" (UniqueName: \"kubernetes.io/projected/daa56966-b092-4927-bac7-11fa18b39d00-kube-api-access-pr7h4\") pod \"ai-gateway-controller-7ddbcdf875-d8wmr\" (UID: \"daa56966-b092-4927-bac7-11fa18b39d00\") " pod="opendatahub/ai-gateway-controller-7ddbcdf875-d8wmr" Oct 30 01:25:40.667597 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:40.667562 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daa56966-b092-4927-bac7-11fa18b39d00-metrics-certs\") pod \"ai-gateway-controller-7ddbcdf875-d8wmr\" (UID: \"daa56966-b092-4927-bac7-11fa18b39d00\") " pod="opendatahub/ai-gateway-controller-7ddbcdf875-d8wmr" Oct 30 01:25:40.667744 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:40.667607 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr7h4\" (UniqueName: \"kubernetes.io/projected/daa56966-b092-4927-bac7-11fa18b39d00-kube-api-access-pr7h4\") pod \"ai-gateway-controller-7ddbcdf875-d8wmr\" (UID: \"daa56966-b092-4927-bac7-11fa18b39d00\") " pod="opendatahub/ai-gateway-controller-7ddbcdf875-d8wmr" Oct 30 01:25:40.670122 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:40.670089 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/daa56966-b092-4927-bac7-11fa18b39d00-metrics-certs\") pod \"ai-gateway-controller-7ddbcdf875-d8wmr\" (UID: \"daa56966-b092-4927-bac7-11fa18b39d00\") " pod="opendatahub/ai-gateway-controller-7ddbcdf875-d8wmr" Oct 30 01:25:40.687056 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:40.687029 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr7h4\" (UniqueName: \"kubernetes.io/projected/daa56966-b092-4927-bac7-11fa18b39d00-kube-api-access-pr7h4\") pod \"ai-gateway-controller-7ddbcdf875-d8wmr\" (UID: \"daa56966-b092-4927-bac7-11fa18b39d00\") " pod="opendatahub/ai-gateway-controller-7ddbcdf875-d8wmr" Oct 30 01:25:40.837758 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:40.837724 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7ddbcdf875-d8wmr" Oct 30 01:25:41.129033 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:41.128942 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7ddbcdf875-d8wmr"] Oct 30 01:25:41.133821 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:25:41.133789 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaa56966_b092_4927_bac7_11fa18b39d00.slice/crio-c3829358284f8575cae50a410bfae0f463fffaf057a1bd2b7a433197676c6348 WatchSource:0}: Error finding container c3829358284f8575cae50a410bfae0f463fffaf057a1bd2b7a433197676c6348: Status 404 returned error can't find the container with id c3829358284f8575cae50a410bfae0f463fffaf057a1bd2b7a433197676c6348 Oct 30 01:25:41.267993 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:41.267947 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7ddbcdf875-d8wmr" event={"ID":"daa56966-b092-4927-bac7-11fa18b39d00","Type":"ContainerStarted","Data":"c3829358284f8575cae50a410bfae0f463fffaf057a1bd2b7a433197676c6348"} Oct 30 01:25:42.273472 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:42.273434 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7ddbcdf875-d8wmr" event={"ID":"daa56966-b092-4927-bac7-11fa18b39d00","Type":"ContainerStarted","Data":"8d411b26c5d68329ca017764e6f74282d8b9c0e9f7cee97ac70a66492bb53c9c"} Oct 30 01:25:42.273888 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:42.273516 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7ddbcdf875-d8wmr" Oct 30 01:25:42.311874 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:42.311828 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7ddbcdf875-d8wmr" podStartSLOduration=1.9614041850000001 podStartE2EDuration="2.311814845s" totalImagesPullingTime="350.41066ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:25:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:25:42.311155723 +0000 UTC m=+465.889443058" watchObservedRunningTime="2026-10-30 01:25:42.311814845 +0000 UTC m=+465.890102180" Oct 30 01:25:44.244379 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:44.244351 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-wp6mf" Oct 30 01:25:44.244769 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:44.244423 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-728tk" Oct 30 01:25:51.492759 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:25:51.492721 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:25:53.280019 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:53.279988 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7ddbcdf875-d8wmr" Oct 30 01:25:53.343345 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:53.343306 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-dn6pg"] Oct 30 01:25:53.343613 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:53.343557 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-dn6pg" podUID="219e3edc-8211-4364-92ee-203bfb1e914e" containerName="manager" containerID="cri-o://740f9023f4fda8df6904e3f71f7a73f6d3c15a3fe27482f8db774437135733af" gracePeriod=30 Oct 30 01:25:53.568788 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:53.568763 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-dn6pg" Oct 30 01:25:53.678006 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:53.677972 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/219e3edc-8211-4364-92ee-203bfb1e914e-kube-api-access-tzdpv\" (UniqueName: \"kubernetes.io/projected/219e3edc-8211-4364-92ee-203bfb1e914e-kube-api-access-tzdpv\") pod \"219e3edc-8211-4364-92ee-203bfb1e914e\" (UID: \"219e3edc-8211-4364-92ee-203bfb1e914e\") " Oct 30 01:25:53.678182 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:53.678095 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/219e3edc-8211-4364-92ee-203bfb1e914e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/219e3edc-8211-4364-92ee-203bfb1e914e-metrics-certs\") pod \"219e3edc-8211-4364-92ee-203bfb1e914e\" (UID: \"219e3edc-8211-4364-92ee-203bfb1e914e\") " Oct 30 01:25:53.680240 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:53.680210 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/219e3edc-8211-4364-92ee-203bfb1e914e-metrics-certs" pod "219e3edc-8211-4364-92ee-203bfb1e914e" (UID: "219e3edc-8211-4364-92ee-203bfb1e914e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:25:53.680359 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:53.680285 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/219e3edc-8211-4364-92ee-203bfb1e914e-kube-api-access-tzdpv" pod "219e3edc-8211-4364-92ee-203bfb1e914e" (UID: "219e3edc-8211-4364-92ee-203bfb1e914e"). InnerVolumeSpecName "kube-api-access-tzdpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:25:53.779164 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:53.779129 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzdpv\" (UniqueName: \"kubernetes.io/projected/219e3edc-8211-4364-92ee-203bfb1e914e-kube-api-access-tzdpv\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:25:53.779164 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:53.779159 6221 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/219e3edc-8211-4364-92ee-203bfb1e914e-metrics-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:25:54.325592 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:54.325556 6221 generic.go:356] "Generic (PLEG): container finished" podID="219e3edc-8211-4364-92ee-203bfb1e914e" containerID="740f9023f4fda8df6904e3f71f7a73f6d3c15a3fe27482f8db774437135733af" exitCode=0 Oct 30 01:25:54.326086 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:54.325606 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-dn6pg" event={"ID":"219e3edc-8211-4364-92ee-203bfb1e914e","Type":"ContainerDied","Data":"740f9023f4fda8df6904e3f71f7a73f6d3c15a3fe27482f8db774437135733af"} Oct 30 01:25:54.326086 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:54.325652 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-dn6pg" event={"ID":"219e3edc-8211-4364-92ee-203bfb1e914e","Type":"ContainerDied","Data":"21582b0ebf27545862e6ebd6e36e75e998d65ecec15bade6506a210c6b0ed293"} Oct 30 01:25:54.326086 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:54.325662 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-dn6pg" Oct 30 01:25:54.326086 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:54.325674 6221 scope.go:122] "RemoveContainer" containerID="740f9023f4fda8df6904e3f71f7a73f6d3c15a3fe27482f8db774437135733af" Oct 30 01:25:54.334414 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:54.334395 6221 scope.go:122] "RemoveContainer" containerID="740f9023f4fda8df6904e3f71f7a73f6d3c15a3fe27482f8db774437135733af" Oct 30 01:25:54.334703 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:25:54.334683 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"740f9023f4fda8df6904e3f71f7a73f6d3c15a3fe27482f8db774437135733af\": container with ID starting with 740f9023f4fda8df6904e3f71f7a73f6d3c15a3fe27482f8db774437135733af not found: ID does not exist" containerID="740f9023f4fda8df6904e3f71f7a73f6d3c15a3fe27482f8db774437135733af" Oct 30 01:25:54.334744 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:54.334711 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"740f9023f4fda8df6904e3f71f7a73f6d3c15a3fe27482f8db774437135733af"} err="failed to get container status \"740f9023f4fda8df6904e3f71f7a73f6d3c15a3fe27482f8db774437135733af\": rpc error: code = NotFound desc = could not find container \"740f9023f4fda8df6904e3f71f7a73f6d3c15a3fe27482f8db774437135733af\": container with ID starting with 740f9023f4fda8df6904e3f71f7a73f6d3c15a3fe27482f8db774437135733af not found: ID does not exist" Oct 30 01:25:54.375440 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:54.375407 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-dn6pg"] Oct 30 01:25:54.387787 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:54.387760 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-dn6pg"] Oct 30 01:25:54.549480 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:25:54.546156 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="219e3edc-8211-4364-92ee-203bfb1e914e" path="/var/lib/kubelet/pods/219e3edc-8211-4364-92ee-203bfb1e914e/volumes" Oct 30 01:26:01.108860 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:01.108826 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s2bnf"] Oct 30 01:26:01.109353 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:01.109249 6221 cpu_manager.go:414] "containerMap: removing container" podUID="219e3edc-8211-4364-92ee-203bfb1e914e" containerName="manager" Oct 30 01:26:01.109353 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:01.109266 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="219e3edc-8211-4364-92ee-203bfb1e914e" containerName="manager" Oct 30 01:26:01.109524 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:01.109393 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="219e3edc-8211-4364-92ee-203bfb1e914e" containerName="manager" Oct 30 01:26:01.113711 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:01.113692 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s2bnf" Oct 30 01:26:01.129477 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:01.129436 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 01:26:01.129477 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:01.129461 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 01:26:01.141910 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:01.141885 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-xzxn7\"" Oct 30 01:26:01.142063 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:01.141983 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 30 01:26:01.143018 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:01.142999 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s2bnf"] Oct 30 01:26:01.242808 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:01.242777 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c94e2ef-848a-4530-a644-51419cdb6fbe-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s2bnf\" (UID: \"8c94e2ef-848a-4530-a644-51419cdb6fbe\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s2bnf" Oct 30 01:26:01.243004 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:01.242817 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c94e2ef-848a-4530-a644-51419cdb6fbe-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s2bnf\" (UID: \"8c94e2ef-848a-4530-a644-51419cdb6fbe\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s2bnf" Oct 30 01:26:01.243004 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:01.242837 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48dxr\" (UniqueName: \"kubernetes.io/projected/8c94e2ef-848a-4530-a644-51419cdb6fbe-kube-api-access-48dxr\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s2bnf\" (UID: \"8c94e2ef-848a-4530-a644-51419cdb6fbe\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s2bnf" Oct 30 01:26:01.243004 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:01.242979 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c94e2ef-848a-4530-a644-51419cdb6fbe-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s2bnf\" (UID: \"8c94e2ef-848a-4530-a644-51419cdb6fbe\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s2bnf" Oct 30 01:26:01.243141 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:01.243032 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c94e2ef-848a-4530-a644-51419cdb6fbe-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s2bnf\" (UID: \"8c94e2ef-848a-4530-a644-51419cdb6fbe\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s2bnf" Oct 30 01:26:01.243141 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:01.243108 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c94e2ef-848a-4530-a644-51419cdb6fbe-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s2bnf\" (UID: \"8c94e2ef-848a-4530-a644-51419cdb6fbe\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s2bnf" Oct 30 01:26:01.344053 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:01.344016 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c94e2ef-848a-4530-a644-51419cdb6fbe-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s2bnf\" (UID: \"8c94e2ef-848a-4530-a644-51419cdb6fbe\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s2bnf" Oct 30 01:26:01.344053 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:01.344057 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48dxr\" (UniqueName: \"kubernetes.io/projected/8c94e2ef-848a-4530-a644-51419cdb6fbe-kube-api-access-48dxr\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s2bnf\" (UID: \"8c94e2ef-848a-4530-a644-51419cdb6fbe\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s2bnf" Oct 30 01:26:01.344318 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:01.344084 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c94e2ef-848a-4530-a644-51419cdb6fbe-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s2bnf\" (UID: \"8c94e2ef-848a-4530-a644-51419cdb6fbe\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s2bnf" Oct 30 01:26:01.344318 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:01.344119 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c94e2ef-848a-4530-a644-51419cdb6fbe-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s2bnf\" (UID: \"8c94e2ef-848a-4530-a644-51419cdb6fbe\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s2bnf" Oct 30 01:26:01.344318 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:01.344147 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c94e2ef-848a-4530-a644-51419cdb6fbe-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s2bnf\" (UID: \"8c94e2ef-848a-4530-a644-51419cdb6fbe\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s2bnf" Oct 30 01:26:01.344318 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:01.344187 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c94e2ef-848a-4530-a644-51419cdb6fbe-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s2bnf\" (UID: \"8c94e2ef-848a-4530-a644-51419cdb6fbe\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s2bnf" Oct 30 01:26:01.344619 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:01.344588 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8c94e2ef-848a-4530-a644-51419cdb6fbe-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s2bnf\" (UID: \"8c94e2ef-848a-4530-a644-51419cdb6fbe\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s2bnf" Oct 30 01:26:01.344748 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:01.344642 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8c94e2ef-848a-4530-a644-51419cdb6fbe-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s2bnf\" (UID: \"8c94e2ef-848a-4530-a644-51419cdb6fbe\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s2bnf" Oct 30 01:26:01.344748 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:01.344669 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c94e2ef-848a-4530-a644-51419cdb6fbe-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s2bnf\" (UID: \"8c94e2ef-848a-4530-a644-51419cdb6fbe\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s2bnf" Oct 30 01:26:01.346344 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:01.346317 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8c94e2ef-848a-4530-a644-51419cdb6fbe-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s2bnf\" (UID: \"8c94e2ef-848a-4530-a644-51419cdb6fbe\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s2bnf" Oct 30 01:26:01.346652 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:01.346633 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8c94e2ef-848a-4530-a644-51419cdb6fbe-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s2bnf\" (UID: \"8c94e2ef-848a-4530-a644-51419cdb6fbe\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s2bnf" Oct 30 01:26:01.366273 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:01.366214 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48dxr\" (UniqueName: \"kubernetes.io/projected/8c94e2ef-848a-4530-a644-51419cdb6fbe-kube-api-access-48dxr\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-s2bnf\" (UID: \"8c94e2ef-848a-4530-a644-51419cdb6fbe\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s2bnf" Oct 30 01:26:01.425414 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:01.425358 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s2bnf" Oct 30 01:26:01.719223 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:01.719138 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s2bnf"] Oct 30 01:26:01.723507 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:26:01.723479 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c94e2ef_848a_4530_a644_51419cdb6fbe.slice/crio-b44f4cd3bb1cb4fccde4ee7479e8cbc9018bc79dc3513ee3df454c946b079259 WatchSource:0}: Error finding container b44f4cd3bb1cb4fccde4ee7479e8cbc9018bc79dc3513ee3df454c946b079259: Status 404 returned error can't find the container with id b44f4cd3bb1cb4fccde4ee7479e8cbc9018bc79dc3513ee3df454c946b079259 Oct 30 01:26:02.231577 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:02.231523 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:26:02.360956 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:02.360921 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s2bnf" event={"ID":"8c94e2ef-848a-4530-a644-51419cdb6fbe","Type":"ContainerStarted","Data":"b44f4cd3bb1cb4fccde4ee7479e8cbc9018bc79dc3513ee3df454c946b079259"} Oct 30 01:26:03.365922 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:03.365875 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s2bnf" event={"ID":"8c94e2ef-848a-4530-a644-51419cdb6fbe","Type":"ContainerStarted","Data":"efd3e0add3c572739b8b02238b5f5117fa577bfa67ceb61df9119ffd21dd4c25"} Oct 30 01:26:03.404611 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:03.404561 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s2bnf" podStartSLOduration=1.183865852 podStartE2EDuration="2.404546982s" totalImagesPullingTime="1.22068113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:26:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:26:03.403773001 +0000 UTC m=+486.982060337" watchObservedRunningTime="2026-10-30 01:26:03.404546982 +0000 UTC m=+486.982834315" Oct 30 01:26:03.425678 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:03.425635 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s2bnf" Oct 30 01:26:03.432199 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:03.432168 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-pmwss"] Oct 30 01:26:03.434724 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:03.434704 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pmwss" Oct 30 01:26:03.449579 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:03.449518 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 30 01:26:03.457304 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:03.457277 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-pmwss"] Oct 30 01:26:03.567390 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:03.567348 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94880b9b-8725-4337-aee3-0ec765960248-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pmwss\" (UID: \"94880b9b-8725-4337-aee3-0ec765960248\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pmwss" Oct 30 01:26:03.567390 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:03.567394 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96vmm\" (UniqueName: \"kubernetes.io/projected/94880b9b-8725-4337-aee3-0ec765960248-kube-api-access-96vmm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pmwss\" (UID: \"94880b9b-8725-4337-aee3-0ec765960248\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pmwss" Oct 30 01:26:03.567691 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:03.567420 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94880b9b-8725-4337-aee3-0ec765960248-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pmwss\" (UID: \"94880b9b-8725-4337-aee3-0ec765960248\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pmwss" Oct 30 01:26:03.567691 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:03.567586 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94880b9b-8725-4337-aee3-0ec765960248-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pmwss\" (UID: \"94880b9b-8725-4337-aee3-0ec765960248\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pmwss" Oct 30 01:26:03.567691 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:03.567625 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94880b9b-8725-4337-aee3-0ec765960248-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pmwss\" (UID: \"94880b9b-8725-4337-aee3-0ec765960248\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pmwss" Oct 30 01:26:03.567691 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:03.567659 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94880b9b-8725-4337-aee3-0ec765960248-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pmwss\" (UID: \"94880b9b-8725-4337-aee3-0ec765960248\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pmwss" Oct 30 01:26:03.668982 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:03.668898 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94880b9b-8725-4337-aee3-0ec765960248-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pmwss\" (UID: \"94880b9b-8725-4337-aee3-0ec765960248\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pmwss" Oct 30 01:26:03.668982 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:03.668940 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94880b9b-8725-4337-aee3-0ec765960248-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pmwss\" (UID: \"94880b9b-8725-4337-aee3-0ec765960248\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pmwss" Oct 30 01:26:03.668982 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:03.668960 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94880b9b-8725-4337-aee3-0ec765960248-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pmwss\" (UID: \"94880b9b-8725-4337-aee3-0ec765960248\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pmwss" Oct 30 01:26:03.669271 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:03.668991 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94880b9b-8725-4337-aee3-0ec765960248-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pmwss\" (UID: \"94880b9b-8725-4337-aee3-0ec765960248\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pmwss" Oct 30 01:26:03.669271 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:03.669083 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94880b9b-8725-4337-aee3-0ec765960248-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pmwss\" (UID: \"94880b9b-8725-4337-aee3-0ec765960248\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pmwss" Oct 30 01:26:03.669271 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:03.669115 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96vmm\" (UniqueName: \"kubernetes.io/projected/94880b9b-8725-4337-aee3-0ec765960248-kube-api-access-96vmm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pmwss\" (UID: \"94880b9b-8725-4337-aee3-0ec765960248\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pmwss" Oct 30 01:26:03.669453 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:03.669356 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94880b9b-8725-4337-aee3-0ec765960248-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pmwss\" (UID: \"94880b9b-8725-4337-aee3-0ec765960248\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pmwss" Oct 30 01:26:03.669453 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:03.669376 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94880b9b-8725-4337-aee3-0ec765960248-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pmwss\" (UID: \"94880b9b-8725-4337-aee3-0ec765960248\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pmwss" Oct 30 01:26:03.669573 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:03.669492 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94880b9b-8725-4337-aee3-0ec765960248-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pmwss\" (UID: \"94880b9b-8725-4337-aee3-0ec765960248\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pmwss" Oct 30 01:26:03.671353 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:03.671327 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94880b9b-8725-4337-aee3-0ec765960248-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pmwss\" (UID: \"94880b9b-8725-4337-aee3-0ec765960248\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pmwss" Oct 30 01:26:03.671609 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:03.671592 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94880b9b-8725-4337-aee3-0ec765960248-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pmwss\" (UID: \"94880b9b-8725-4337-aee3-0ec765960248\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pmwss" Oct 30 01:26:03.688613 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:03.688585 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96vmm\" (UniqueName: \"kubernetes.io/projected/94880b9b-8725-4337-aee3-0ec765960248-kube-api-access-96vmm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-pmwss\" (UID: \"94880b9b-8725-4337-aee3-0ec765960248\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pmwss" Oct 30 01:26:03.746633 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:03.746596 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pmwss" Oct 30 01:26:04.039966 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:04.039932 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-pmwss"] Oct 30 01:26:04.044012 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:26:04.043983 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94880b9b_8725_4337_aee3_0ec765960248.slice/crio-cb3909da00cd0489a13185ee9fd5b85f6313da9793c8214698513e91a829c84a WatchSource:0}: Error finding container cb3909da00cd0489a13185ee9fd5b85f6313da9793c8214698513e91a829c84a: Status 404 returned error can't find the container with id cb3909da00cd0489a13185ee9fd5b85f6313da9793c8214698513e91a829c84a Oct 30 01:26:04.371138 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:04.371038 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pmwss" event={"ID":"94880b9b-8725-4337-aee3-0ec765960248","Type":"ContainerStarted","Data":"3fd6663e80d267a498978c3244e897eb0e202cb1597c94a2b1932ed8847a7a0d"} Oct 30 01:26:04.371138 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:04.371103 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pmwss" event={"ID":"94880b9b-8725-4337-aee3-0ec765960248","Type":"ContainerStarted","Data":"cb3909da00cd0489a13185ee9fd5b85f6313da9793c8214698513e91a829c84a"} Oct 30 01:26:04.411134 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:04.411075 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pmwss" podStartSLOduration=1.217499552 podStartE2EDuration="1.411056305s" totalImagesPullingTime="193.556753ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:26:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:26:04.410224251 +0000 UTC m=+487.988511587" watchObservedRunningTime="2026-10-30 01:26:04.411056305 +0000 UTC m=+487.989343640" Oct 30 01:26:04.747871 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:04.747772 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pmwss" Oct 30 01:26:06.186312 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:06.186273 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-22l7k"] Oct 30 01:26:06.189281 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:06.189259 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-22l7k" Oct 30 01:26:06.199330 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:06.199302 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:26:06.209523 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:06.209461 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-22l7k"] Oct 30 01:26:06.210327 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:06.210300 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 30 01:26:06.289776 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:06.289736 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f3def97-cd3e-4553-abc8-35cbd888b408-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-22l7k\" (UID: \"1f3def97-cd3e-4553-abc8-35cbd888b408\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-22l7k" Oct 30 01:26:06.289966 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:06.289792 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f3def97-cd3e-4553-abc8-35cbd888b408-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-22l7k\" (UID: \"1f3def97-cd3e-4553-abc8-35cbd888b408\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-22l7k" Oct 30 01:26:06.289966 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:06.289919 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl4gp\" (UniqueName: \"kubernetes.io/projected/1f3def97-cd3e-4553-abc8-35cbd888b408-kube-api-access-bl4gp\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-22l7k\" (UID: \"1f3def97-cd3e-4553-abc8-35cbd888b408\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-22l7k" Oct 30 01:26:06.289966 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:06.289957 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f3def97-cd3e-4553-abc8-35cbd888b408-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-22l7k\" (UID: \"1f3def97-cd3e-4553-abc8-35cbd888b408\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-22l7k" Oct 30 01:26:06.290137 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:06.289979 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f3def97-cd3e-4553-abc8-35cbd888b408-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-22l7k\" (UID: \"1f3def97-cd3e-4553-abc8-35cbd888b408\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-22l7k" Oct 30 01:26:06.290137 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:06.290028 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f3def97-cd3e-4553-abc8-35cbd888b408-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-22l7k\" (UID: \"1f3def97-cd3e-4553-abc8-35cbd888b408\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-22l7k" Oct 30 01:26:06.391092 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:06.391049 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl4gp\" (UniqueName: \"kubernetes.io/projected/1f3def97-cd3e-4553-abc8-35cbd888b408-kube-api-access-bl4gp\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-22l7k\" (UID: \"1f3def97-cd3e-4553-abc8-35cbd888b408\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-22l7k" Oct 30 01:26:06.391268 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:06.391113 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f3def97-cd3e-4553-abc8-35cbd888b408-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-22l7k\" (UID: \"1f3def97-cd3e-4553-abc8-35cbd888b408\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-22l7k" Oct 30 01:26:06.391268 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:06.391148 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f3def97-cd3e-4553-abc8-35cbd888b408-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-22l7k\" (UID: \"1f3def97-cd3e-4553-abc8-35cbd888b408\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-22l7k" Oct 30 01:26:06.391268 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:06.391188 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f3def97-cd3e-4553-abc8-35cbd888b408-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-22l7k\" (UID: \"1f3def97-cd3e-4553-abc8-35cbd888b408\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-22l7k" Oct 30 01:26:06.391268 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:06.391225 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f3def97-cd3e-4553-abc8-35cbd888b408-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-22l7k\" (UID: \"1f3def97-cd3e-4553-abc8-35cbd888b408\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-22l7k" Oct 30 01:26:06.391507 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:06.391274 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f3def97-cd3e-4553-abc8-35cbd888b408-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-22l7k\" (UID: \"1f3def97-cd3e-4553-abc8-35cbd888b408\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-22l7k" Oct 30 01:26:06.391643 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:06.391615 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f3def97-cd3e-4553-abc8-35cbd888b408-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-22l7k\" (UID: \"1f3def97-cd3e-4553-abc8-35cbd888b408\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-22l7k" Oct 30 01:26:06.391643 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:06.391633 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f3def97-cd3e-4553-abc8-35cbd888b408-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-22l7k\" (UID: \"1f3def97-cd3e-4553-abc8-35cbd888b408\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-22l7k" Oct 30 01:26:06.391959 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:06.391940 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f3def97-cd3e-4553-abc8-35cbd888b408-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-22l7k\" (UID: \"1f3def97-cd3e-4553-abc8-35cbd888b408\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-22l7k" Oct 30 01:26:06.393621 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:06.393597 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f3def97-cd3e-4553-abc8-35cbd888b408-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-22l7k\" (UID: \"1f3def97-cd3e-4553-abc8-35cbd888b408\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-22l7k" Oct 30 01:26:06.393774 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:06.393756 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f3def97-cd3e-4553-abc8-35cbd888b408-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-22l7k\" (UID: \"1f3def97-cd3e-4553-abc8-35cbd888b408\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-22l7k" Oct 30 01:26:06.410790 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:06.410759 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl4gp\" (UniqueName: \"kubernetes.io/projected/1f3def97-cd3e-4553-abc8-35cbd888b408-kube-api-access-bl4gp\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-22l7k\" (UID: \"1f3def97-cd3e-4553-abc8-35cbd888b408\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-22l7k" Oct 30 01:26:06.504302 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:06.504215 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-22l7k" Oct 30 01:26:06.798851 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:06.798816 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-22l7k"] Oct 30 01:26:06.802633 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:26:06.802611 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f3def97_cd3e_4553_abc8_35cbd888b408.slice/crio-c08f7f7a2b0c842c3f3f79e1731276d04dcb94c33f2ff6247ccbe1df653ad128 WatchSource:0}: Error finding container c08f7f7a2b0c842c3f3f79e1731276d04dcb94c33f2ff6247ccbe1df653ad128: Status 404 returned error can't find the container with id c08f7f7a2b0c842c3f3f79e1731276d04dcb94c33f2ff6247ccbe1df653ad128 Oct 30 01:26:07.386078 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:07.386044 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-22l7k" event={"ID":"1f3def97-cd3e-4553-abc8-35cbd888b408","Type":"ContainerStarted","Data":"a866e26bcf308b56bbf74fee71dcebfaa74a90ca3a4ff3c8dcd419b823f3c078"} Oct 30 01:26:07.386446 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:07.386095 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-22l7k" event={"ID":"1f3def97-cd3e-4553-abc8-35cbd888b408","Type":"ContainerStarted","Data":"c08f7f7a2b0c842c3f3f79e1731276d04dcb94c33f2ff6247ccbe1df653ad128"} Oct 30 01:26:07.425025 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:07.424975 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-22l7k" podStartSLOduration=1.234178662 podStartE2EDuration="1.424962513s" totalImagesPullingTime="190.783851ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:26:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:26:07.424149945 +0000 UTC m=+491.002437281" watchObservedRunningTime="2026-10-30 01:26:07.424962513 +0000 UTC m=+491.003249849" Oct 30 01:26:07.504612 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:07.504583 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-22l7k" Oct 30 01:26:08.003713 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:08.003676 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ztcpp"] Oct 30 01:26:08.006057 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:08.006040 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ztcpp" Oct 30 01:26:08.025099 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:08.025074 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Oct 30 01:26:08.035190 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:08.035163 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ztcpp"] Oct 30 01:26:08.104903 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:08.104874 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34f45b5e-e31f-4309-92d4-039b539e1578-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ztcpp\" (UID: \"34f45b5e-e31f-4309-92d4-039b539e1578\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ztcpp" Oct 30 01:26:08.105070 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:08.104912 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34f45b5e-e31f-4309-92d4-039b539e1578-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ztcpp\" (UID: \"34f45b5e-e31f-4309-92d4-039b539e1578\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ztcpp" Oct 30 01:26:08.105070 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:08.104940 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34f45b5e-e31f-4309-92d4-039b539e1578-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ztcpp\" (UID: \"34f45b5e-e31f-4309-92d4-039b539e1578\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ztcpp" Oct 30 01:26:08.105070 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:08.105010 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34f45b5e-e31f-4309-92d4-039b539e1578-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ztcpp\" (UID: \"34f45b5e-e31f-4309-92d4-039b539e1578\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ztcpp" Oct 30 01:26:08.105197 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:08.105072 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5smq\" (UniqueName: \"kubernetes.io/projected/34f45b5e-e31f-4309-92d4-039b539e1578-kube-api-access-d5smq\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ztcpp\" (UID: \"34f45b5e-e31f-4309-92d4-039b539e1578\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ztcpp" Oct 30 01:26:08.105197 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:08.105123 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34f45b5e-e31f-4309-92d4-039b539e1578-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ztcpp\" (UID: \"34f45b5e-e31f-4309-92d4-039b539e1578\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ztcpp" Oct 30 01:26:08.206368 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:08.206337 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34f45b5e-e31f-4309-92d4-039b539e1578-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ztcpp\" (UID: \"34f45b5e-e31f-4309-92d4-039b539e1578\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ztcpp" Oct 30 01:26:08.206543 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:08.206380 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34f45b5e-e31f-4309-92d4-039b539e1578-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ztcpp\" (UID: \"34f45b5e-e31f-4309-92d4-039b539e1578\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ztcpp" Oct 30 01:26:08.206543 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:08.206398 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34f45b5e-e31f-4309-92d4-039b539e1578-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ztcpp\" (UID: \"34f45b5e-e31f-4309-92d4-039b539e1578\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ztcpp" Oct 30 01:26:08.206543 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:08.206427 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5smq\" (UniqueName: \"kubernetes.io/projected/34f45b5e-e31f-4309-92d4-039b539e1578-kube-api-access-d5smq\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ztcpp\" (UID: \"34f45b5e-e31f-4309-92d4-039b539e1578\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ztcpp" Oct 30 01:26:08.206543 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:08.206445 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34f45b5e-e31f-4309-92d4-039b539e1578-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ztcpp\" (UID: \"34f45b5e-e31f-4309-92d4-039b539e1578\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ztcpp" Oct 30 01:26:08.206543 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:08.206483 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34f45b5e-e31f-4309-92d4-039b539e1578-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ztcpp\" (UID: \"34f45b5e-e31f-4309-92d4-039b539e1578\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ztcpp" Oct 30 01:26:08.206903 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:08.206878 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34f45b5e-e31f-4309-92d4-039b539e1578-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ztcpp\" (UID: \"34f45b5e-e31f-4309-92d4-039b539e1578\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ztcpp" Oct 30 01:26:08.206980 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:08.206932 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34f45b5e-e31f-4309-92d4-039b539e1578-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ztcpp\" (UID: \"34f45b5e-e31f-4309-92d4-039b539e1578\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ztcpp" Oct 30 01:26:08.206980 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:08.206949 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34f45b5e-e31f-4309-92d4-039b539e1578-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ztcpp\" (UID: \"34f45b5e-e31f-4309-92d4-039b539e1578\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ztcpp" Oct 30 01:26:08.208675 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:08.208655 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34f45b5e-e31f-4309-92d4-039b539e1578-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ztcpp\" (UID: \"34f45b5e-e31f-4309-92d4-039b539e1578\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ztcpp" Oct 30 01:26:08.209095 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:08.209074 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34f45b5e-e31f-4309-92d4-039b539e1578-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ztcpp\" (UID: \"34f45b5e-e31f-4309-92d4-039b539e1578\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ztcpp" Oct 30 01:26:08.225667 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:08.225641 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5smq\" (UniqueName: \"kubernetes.io/projected/34f45b5e-e31f-4309-92d4-039b539e1578-kube-api-access-d5smq\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ztcpp\" (UID: \"34f45b5e-e31f-4309-92d4-039b539e1578\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ztcpp" Oct 30 01:26:08.320424 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:08.320393 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ztcpp" Oct 30 01:26:08.611864 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:08.611782 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ztcpp"] Oct 30 01:26:08.615292 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:26:08.615264 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34f45b5e_e31f_4309_92d4_039b539e1578.slice/crio-111bf3ba560c4446483ed201676291799768291766cc1aa67f270d8ef7d06247 WatchSource:0}: Error finding container 111bf3ba560c4446483ed201676291799768291766cc1aa67f270d8ef7d06247: Status 404 returned error can't find the container with id 111bf3ba560c4446483ed201676291799768291766cc1aa67f270d8ef7d06247 Oct 30 01:26:09.396406 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:09.396358 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ztcpp" event={"ID":"34f45b5e-e31f-4309-92d4-039b539e1578","Type":"ContainerStarted","Data":"0cb2af6cdb2b0f0b1c6454bbcdbc166d01e233cc5df1c9f43089b4cd1eaf6906"} Oct 30 01:26:09.396599 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:09.396453 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ztcpp" event={"ID":"34f45b5e-e31f-4309-92d4-039b539e1578","Type":"ContainerStarted","Data":"111bf3ba560c4446483ed201676291799768291766cc1aa67f270d8ef7d06247"} Oct 30 01:26:09.439093 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:09.439045 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ztcpp" podStartSLOduration=2.233100029 podStartE2EDuration="2.43903334s" totalImagesPullingTime="205.933311ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:26:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:26:09.438564057 +0000 UTC m=+493.016851392" watchObservedRunningTime="2026-10-30 01:26:09.43903334 +0000 UTC m=+493.017320719" Oct 30 01:26:10.320782 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:10.320749 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ztcpp" Oct 30 01:26:11.426034 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:11.426002 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s2bnf" Oct 30 01:26:11.438600 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:11.438574 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s2bnf" Oct 30 01:26:12.421983 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:12.421957 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-s2bnf" Oct 30 01:26:12.716796 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:12.716711 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:26:13.747013 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:13.746981 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pmwss" Oct 30 01:26:13.759390 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:13.759365 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pmwss" Oct 30 01:26:14.431394 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:14.431362 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-pmwss" Oct 30 01:26:16.054061 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:16.054028 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk"] Oct 30 01:26:16.056463 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:16.056442 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk" Oct 30 01:26:16.083814 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:16.083792 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 30 01:26:16.085098 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:16.085077 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk"] Oct 30 01:26:16.177878 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:16.177837 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4g55\" (UniqueName: \"kubernetes.io/projected/f4ae494e-81ec-4039-ae07-7ed1bf345f0f-kube-api-access-v4g55\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk\" (UID: \"f4ae494e-81ec-4039-ae07-7ed1bf345f0f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk" Oct 30 01:26:16.177878 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:16.177886 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4ae494e-81ec-4039-ae07-7ed1bf345f0f-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk\" (UID: \"f4ae494e-81ec-4039-ae07-7ed1bf345f0f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk" Oct 30 01:26:16.178097 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:16.178013 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4ae494e-81ec-4039-ae07-7ed1bf345f0f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk\" (UID: \"f4ae494e-81ec-4039-ae07-7ed1bf345f0f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk" Oct 30 01:26:16.178097 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:16.178061 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4ae494e-81ec-4039-ae07-7ed1bf345f0f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk\" (UID: \"f4ae494e-81ec-4039-ae07-7ed1bf345f0f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk" Oct 30 01:26:16.178097 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:16.178091 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4ae494e-81ec-4039-ae07-7ed1bf345f0f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk\" (UID: \"f4ae494e-81ec-4039-ae07-7ed1bf345f0f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk" Oct 30 01:26:16.178241 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:16.178206 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4ae494e-81ec-4039-ae07-7ed1bf345f0f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk\" (UID: \"f4ae494e-81ec-4039-ae07-7ed1bf345f0f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk" Oct 30 01:26:16.279069 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:16.279032 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4g55\" (UniqueName: \"kubernetes.io/projected/f4ae494e-81ec-4039-ae07-7ed1bf345f0f-kube-api-access-v4g55\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk\" (UID: \"f4ae494e-81ec-4039-ae07-7ed1bf345f0f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk" Oct 30 01:26:16.279069 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:16.279071 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4ae494e-81ec-4039-ae07-7ed1bf345f0f-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk\" (UID: \"f4ae494e-81ec-4039-ae07-7ed1bf345f0f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk" Oct 30 01:26:16.279270 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:16.279109 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4ae494e-81ec-4039-ae07-7ed1bf345f0f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk\" (UID: \"f4ae494e-81ec-4039-ae07-7ed1bf345f0f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk" Oct 30 01:26:16.279270 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:16.279131 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4ae494e-81ec-4039-ae07-7ed1bf345f0f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk\" (UID: \"f4ae494e-81ec-4039-ae07-7ed1bf345f0f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk" Oct 30 01:26:16.279270 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:16.279151 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4ae494e-81ec-4039-ae07-7ed1bf345f0f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk\" (UID: \"f4ae494e-81ec-4039-ae07-7ed1bf345f0f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk" Oct 30 01:26:16.279270 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:16.279168 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4ae494e-81ec-4039-ae07-7ed1bf345f0f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk\" (UID: \"f4ae494e-81ec-4039-ae07-7ed1bf345f0f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk" Oct 30 01:26:16.279553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:16.279516 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4ae494e-81ec-4039-ae07-7ed1bf345f0f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk\" (UID: \"f4ae494e-81ec-4039-ae07-7ed1bf345f0f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk" Oct 30 01:26:16.279600 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:16.279518 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4ae494e-81ec-4039-ae07-7ed1bf345f0f-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk\" (UID: \"f4ae494e-81ec-4039-ae07-7ed1bf345f0f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk" Oct 30 01:26:16.279600 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:16.279584 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4ae494e-81ec-4039-ae07-7ed1bf345f0f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk\" (UID: \"f4ae494e-81ec-4039-ae07-7ed1bf345f0f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk" Oct 30 01:26:16.281368 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:16.281342 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4ae494e-81ec-4039-ae07-7ed1bf345f0f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk\" (UID: \"f4ae494e-81ec-4039-ae07-7ed1bf345f0f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk" Oct 30 01:26:16.281713 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:16.281694 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4ae494e-81ec-4039-ae07-7ed1bf345f0f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk\" (UID: \"f4ae494e-81ec-4039-ae07-7ed1bf345f0f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk" Oct 30 01:26:16.297883 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:16.297857 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4g55\" (UniqueName: \"kubernetes.io/projected/f4ae494e-81ec-4039-ae07-7ed1bf345f0f-kube-api-access-v4g55\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk\" (UID: \"f4ae494e-81ec-4039-ae07-7ed1bf345f0f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk" Oct 30 01:26:16.367968 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:16.367892 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk" Oct 30 01:26:16.505158 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:16.505120 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-22l7k" Oct 30 01:26:16.517380 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:16.517359 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-22l7k" Oct 30 01:26:16.660626 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:16.660524 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk"] Oct 30 01:26:16.664722 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:26:16.664693 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4ae494e_81ec_4039_ae07_7ed1bf345f0f.slice/crio-af86d5e7fc77362ce7145518af0a7083ae59a4dd02159785f35a6acd38119905 WatchSource:0}: Error finding container af86d5e7fc77362ce7145518af0a7083ae59a4dd02159785f35a6acd38119905: Status 404 returned error can't find the container with id af86d5e7fc77362ce7145518af0a7083ae59a4dd02159785f35a6acd38119905 Oct 30 01:26:17.436181 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:17.436143 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk" event={"ID":"f4ae494e-81ec-4039-ae07-7ed1bf345f0f","Type":"ContainerStarted","Data":"50e8cebe080d4151294c6cd461225d12da44291e9af7ddc569e0db20b6d35985"} Oct 30 01:26:17.436622 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:17.436192 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk" event={"ID":"f4ae494e-81ec-4039-ae07-7ed1bf345f0f","Type":"ContainerStarted","Data":"af86d5e7fc77362ce7145518af0a7083ae59a4dd02159785f35a6acd38119905"} Oct 30 01:26:17.447078 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:17.447058 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-22l7k" Oct 30 01:26:17.475258 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:17.475213 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk" podStartSLOduration=1.283287914 podStartE2EDuration="1.475201074s" totalImagesPullingTime="191.91316ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:26:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:26:17.474095134 +0000 UTC m=+501.052382471" watchObservedRunningTime="2026-10-30 01:26:17.475201074 +0000 UTC m=+501.053488409" Oct 30 01:26:17.908887 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:17.908839 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:26:18.320674 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:18.320635 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ztcpp" Oct 30 01:26:18.333156 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:18.333133 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ztcpp" Oct 30 01:26:18.368482 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:18.368445 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk" Oct 30 01:26:18.455117 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:18.455090 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ztcpp" Oct 30 01:26:21.493065 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:26:21.493032 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:26:26.368497 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:26.368454 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk" Oct 30 01:26:26.381366 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:26.381337 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk" Oct 30 01:26:26.488630 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:26.488600 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk" Oct 30 01:26:28.326323 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:28.326292 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b"] Oct 30 01:26:28.333189 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:28.333165 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b" Oct 30 01:26:28.363042 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:28.363012 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-s9nfd\"" Oct 30 01:26:28.364262 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:28.364235 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b"] Oct 30 01:26:28.390917 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:28.390890 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f33783d9-0aa2-4089-878d-ffc93fded2c8-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wvm5b\" (UID: \"f33783d9-0aa2-4089-878d-ffc93fded2c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b" Oct 30 01:26:28.391068 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:28.390921 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f33783d9-0aa2-4089-878d-ffc93fded2c8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wvm5b\" (UID: \"f33783d9-0aa2-4089-878d-ffc93fded2c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b" Oct 30 01:26:28.391068 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:28.390948 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f33783d9-0aa2-4089-878d-ffc93fded2c8-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wvm5b\" (UID: \"f33783d9-0aa2-4089-878d-ffc93fded2c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b" Oct 30 01:26:28.391068 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:28.391006 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f33783d9-0aa2-4089-878d-ffc93fded2c8-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wvm5b\" (UID: \"f33783d9-0aa2-4089-878d-ffc93fded2c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b" Oct 30 01:26:28.391225 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:28.391075 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f33783d9-0aa2-4089-878d-ffc93fded2c8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wvm5b\" (UID: \"f33783d9-0aa2-4089-878d-ffc93fded2c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b" Oct 30 01:26:28.391225 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:28.391113 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f33783d9-0aa2-4089-878d-ffc93fded2c8-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wvm5b\" (UID: \"f33783d9-0aa2-4089-878d-ffc93fded2c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b" Oct 30 01:26:28.391225 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:28.391148 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f33783d9-0aa2-4089-878d-ffc93fded2c8-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wvm5b\" (UID: \"f33783d9-0aa2-4089-878d-ffc93fded2c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b" Oct 30 01:26:28.391225 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:28.391221 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f33783d9-0aa2-4089-878d-ffc93fded2c8-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wvm5b\" (UID: \"f33783d9-0aa2-4089-878d-ffc93fded2c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b" Oct 30 01:26:28.391350 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:28.391254 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgzsj\" (UniqueName: \"kubernetes.io/projected/f33783d9-0aa2-4089-878d-ffc93fded2c8-kube-api-access-fgzsj\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wvm5b\" (UID: \"f33783d9-0aa2-4089-878d-ffc93fded2c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b" Oct 30 01:26:28.492044 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:28.492015 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f33783d9-0aa2-4089-878d-ffc93fded2c8-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wvm5b\" (UID: \"f33783d9-0aa2-4089-878d-ffc93fded2c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b" Oct 30 01:26:28.492217 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:28.492056 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgzsj\" (UniqueName: \"kubernetes.io/projected/f33783d9-0aa2-4089-878d-ffc93fded2c8-kube-api-access-fgzsj\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wvm5b\" (UID: \"f33783d9-0aa2-4089-878d-ffc93fded2c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b" Oct 30 01:26:28.492217 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:28.492106 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f33783d9-0aa2-4089-878d-ffc93fded2c8-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wvm5b\" (UID: \"f33783d9-0aa2-4089-878d-ffc93fded2c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b" Oct 30 01:26:28.492217 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:28.492121 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f33783d9-0aa2-4089-878d-ffc93fded2c8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wvm5b\" (UID: \"f33783d9-0aa2-4089-878d-ffc93fded2c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b" Oct 30 01:26:28.492217 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:28.492148 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f33783d9-0aa2-4089-878d-ffc93fded2c8-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wvm5b\" (UID: \"f33783d9-0aa2-4089-878d-ffc93fded2c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b" Oct 30 01:26:28.492217 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:28.492165 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f33783d9-0aa2-4089-878d-ffc93fded2c8-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wvm5b\" (UID: \"f33783d9-0aa2-4089-878d-ffc93fded2c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b" Oct 30 01:26:28.492217 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:28.492193 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f33783d9-0aa2-4089-878d-ffc93fded2c8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wvm5b\" (UID: \"f33783d9-0aa2-4089-878d-ffc93fded2c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b" Oct 30 01:26:28.492641 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:28.492220 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f33783d9-0aa2-4089-878d-ffc93fded2c8-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wvm5b\" (UID: \"f33783d9-0aa2-4089-878d-ffc93fded2c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b" Oct 30 01:26:28.492641 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:28.492250 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f33783d9-0aa2-4089-878d-ffc93fded2c8-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wvm5b\" (UID: \"f33783d9-0aa2-4089-878d-ffc93fded2c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b" Oct 30 01:26:28.492641 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:28.492571 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f33783d9-0aa2-4089-878d-ffc93fded2c8-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wvm5b\" (UID: \"f33783d9-0aa2-4089-878d-ffc93fded2c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b" Oct 30 01:26:28.492813 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:28.492635 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f33783d9-0aa2-4089-878d-ffc93fded2c8-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wvm5b\" (UID: \"f33783d9-0aa2-4089-878d-ffc93fded2c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b" Oct 30 01:26:28.492909 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:28.492890 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f33783d9-0aa2-4089-878d-ffc93fded2c8-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wvm5b\" (UID: \"f33783d9-0aa2-4089-878d-ffc93fded2c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b" Oct 30 01:26:28.492971 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:28.492942 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f33783d9-0aa2-4089-878d-ffc93fded2c8-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wvm5b\" (UID: \"f33783d9-0aa2-4089-878d-ffc93fded2c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b" Oct 30 01:26:28.492971 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:28.492952 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f33783d9-0aa2-4089-878d-ffc93fded2c8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wvm5b\" (UID: \"f33783d9-0aa2-4089-878d-ffc93fded2c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b" Oct 30 01:26:28.494395 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:28.494374 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f33783d9-0aa2-4089-878d-ffc93fded2c8-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wvm5b\" (UID: \"f33783d9-0aa2-4089-878d-ffc93fded2c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b" Oct 30 01:26:28.494655 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:28.494636 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f33783d9-0aa2-4089-878d-ffc93fded2c8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wvm5b\" (UID: \"f33783d9-0aa2-4089-878d-ffc93fded2c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b" Oct 30 01:26:28.513323 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:28.513301 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f33783d9-0aa2-4089-878d-ffc93fded2c8-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wvm5b\" (UID: \"f33783d9-0aa2-4089-878d-ffc93fded2c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b" Oct 30 01:26:28.513705 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:28.513688 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgzsj\" (UniqueName: \"kubernetes.io/projected/f33783d9-0aa2-4089-878d-ffc93fded2c8-kube-api-access-fgzsj\") pod \"maas-default-gateway-openshift-default-5bc884dd87-wvm5b\" (UID: \"f33783d9-0aa2-4089-878d-ffc93fded2c8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b" Oct 30 01:26:28.646295 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:28.646216 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b" Oct 30 01:26:28.956099 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:28.956024 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b"] Oct 30 01:26:28.960613 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:26:28.960586 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf33783d9_0aa2_4089_878d_ffc93fded2c8.slice/crio-ed404d7c83c9f2507744a3a8568326b6fc3a6623d821881e938eab3e391b4ab6 WatchSource:0}: Error finding container ed404d7c83c9f2507744a3a8568326b6fc3a6623d821881e938eab3e391b4ab6: Status 404 returned error can't find the container with id ed404d7c83c9f2507744a3a8568326b6fc3a6623d821881e938eab3e391b4ab6 Oct 30 01:26:28.962577 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:28.962557 6221 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:26:29.491324 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:29.491288 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b" event={"ID":"f33783d9-0aa2-4089-878d-ffc93fded2c8","Type":"ContainerStarted","Data":"ed404d7c83c9f2507744a3a8568326b6fc3a6623d821881e938eab3e391b4ab6"} Oct 30 01:26:32.164459 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:32.164427 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:26:32.164779 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:32.164494 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:26:32.164779 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:32.164522 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:26:32.507973 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:32.507881 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b" event={"ID":"f33783d9-0aa2-4089-878d-ffc93fded2c8","Type":"ContainerStarted","Data":"834aefe8b3df55015510da5c1c63f2ab13cab3ad189129004881e6deef12f73c"} Oct 30 01:26:32.563899 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:32.562938 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b" podStartSLOduration=1.362115835 podStartE2EDuration="4.562918794s" totalImagesPullingTime="3.200802959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:26:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:26:32.561979588 +0000 UTC m=+516.140266918" watchObservedRunningTime="2026-10-30 01:26:32.562918794 +0000 UTC m=+516.141206130" Oct 30 01:26:32.647172 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:32.647135 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b" Oct 30 01:26:33.648096 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:33.648063 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-wvm5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 01:26:33.648582 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:33.648119 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b" podUID="f33783d9-0aa2-4089-878d-ffc93fded2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 01:26:34.646961 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:34.646927 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-wvm5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 01:26:34.647177 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:34.646986 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b" podUID="f33783d9-0aa2-4089-878d-ffc93fded2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 01:26:35.647328 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:35.647288 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-wvm5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 01:26:35.647844 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:35.647366 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b" podUID="f33783d9-0aa2-4089-878d-ffc93fded2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 01:26:36.647360 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:36.647320 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-wvm5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 01:26:36.647793 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:36.647377 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b" podUID="f33783d9-0aa2-4089-878d-ffc93fded2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 01:26:37.647106 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:37.647060 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-wvm5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 01:26:37.647294 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:37.647120 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b" podUID="f33783d9-0aa2-4089-878d-ffc93fded2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 01:26:38.646828 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:38.646794 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-wvm5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 01:26:38.647228 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:38.646869 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b" podUID="f33783d9-0aa2-4089-878d-ffc93fded2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 01:26:40.309611 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:40.309577 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b" Oct 30 01:26:40.310546 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:40.310510 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b" Oct 30 01:26:40.311391 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:40.311370 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b" Oct 30 01:26:43.395194 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:43.395159 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4"] Oct 30 01:26:43.398661 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:43.398643 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4" Oct 30 01:26:43.429942 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:43.429915 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4"] Oct 30 01:26:43.539427 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:43.539393 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39be085b-e445-416b-93c6-52de82dc6a83-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-njbt4\" (UID: \"39be085b-e445-416b-93c6-52de82dc6a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4" Oct 30 01:26:43.539427 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:43.539425 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39be085b-e445-416b-93c6-52de82dc6a83-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-njbt4\" (UID: \"39be085b-e445-416b-93c6-52de82dc6a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4" Oct 30 01:26:43.539664 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:43.539454 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ztpr\" (UniqueName: \"kubernetes.io/projected/39be085b-e445-416b-93c6-52de82dc6a83-kube-api-access-2ztpr\") pod \"maas-default-gateway-openshift-default-5bc884dd87-njbt4\" (UID: \"39be085b-e445-416b-93c6-52de82dc6a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4" Oct 30 01:26:43.539664 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:43.539622 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39be085b-e445-416b-93c6-52de82dc6a83-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-njbt4\" (UID: \"39be085b-e445-416b-93c6-52de82dc6a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4" Oct 30 01:26:43.539734 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:43.539686 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39be085b-e445-416b-93c6-52de82dc6a83-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-njbt4\" (UID: \"39be085b-e445-416b-93c6-52de82dc6a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4" Oct 30 01:26:43.539734 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:43.539718 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39be085b-e445-416b-93c6-52de82dc6a83-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-njbt4\" (UID: \"39be085b-e445-416b-93c6-52de82dc6a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4" Oct 30 01:26:43.539811 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:43.539751 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39be085b-e445-416b-93c6-52de82dc6a83-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-njbt4\" (UID: \"39be085b-e445-416b-93c6-52de82dc6a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4" Oct 30 01:26:43.539811 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:43.539790 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39be085b-e445-416b-93c6-52de82dc6a83-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-njbt4\" (UID: \"39be085b-e445-416b-93c6-52de82dc6a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4" Oct 30 01:26:43.539904 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:43.539846 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39be085b-e445-416b-93c6-52de82dc6a83-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-njbt4\" (UID: \"39be085b-e445-416b-93c6-52de82dc6a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4" Oct 30 01:26:43.640619 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:43.640576 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39be085b-e445-416b-93c6-52de82dc6a83-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-njbt4\" (UID: \"39be085b-e445-416b-93c6-52de82dc6a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4" Oct 30 01:26:43.640800 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:43.640670 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39be085b-e445-416b-93c6-52de82dc6a83-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-njbt4\" (UID: \"39be085b-e445-416b-93c6-52de82dc6a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4" Oct 30 01:26:43.640800 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:43.640705 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39be085b-e445-416b-93c6-52de82dc6a83-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-njbt4\" (UID: \"39be085b-e445-416b-93c6-52de82dc6a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4" Oct 30 01:26:43.640800 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:43.640727 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39be085b-e445-416b-93c6-52de82dc6a83-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-njbt4\" (UID: \"39be085b-e445-416b-93c6-52de82dc6a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4" Oct 30 01:26:43.640800 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:43.640787 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ztpr\" (UniqueName: \"kubernetes.io/projected/39be085b-e445-416b-93c6-52de82dc6a83-kube-api-access-2ztpr\") pod \"maas-default-gateway-openshift-default-5bc884dd87-njbt4\" (UID: \"39be085b-e445-416b-93c6-52de82dc6a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4" Oct 30 01:26:43.641009 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:43.640862 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39be085b-e445-416b-93c6-52de82dc6a83-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-njbt4\" (UID: \"39be085b-e445-416b-93c6-52de82dc6a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4" Oct 30 01:26:43.641009 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:43.640918 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39be085b-e445-416b-93c6-52de82dc6a83-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-njbt4\" (UID: \"39be085b-e445-416b-93c6-52de82dc6a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4" Oct 30 01:26:43.641009 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:43.640988 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39be085b-e445-416b-93c6-52de82dc6a83-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-njbt4\" (UID: \"39be085b-e445-416b-93c6-52de82dc6a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4" Oct 30 01:26:43.641164 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:43.641031 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39be085b-e445-416b-93c6-52de82dc6a83-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-njbt4\" (UID: \"39be085b-e445-416b-93c6-52de82dc6a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4" Oct 30 01:26:43.641164 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:43.641059 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39be085b-e445-416b-93c6-52de82dc6a83-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-njbt4\" (UID: \"39be085b-e445-416b-93c6-52de82dc6a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4" Oct 30 01:26:43.641164 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:43.641079 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39be085b-e445-416b-93c6-52de82dc6a83-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-njbt4\" (UID: \"39be085b-e445-416b-93c6-52de82dc6a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4" Oct 30 01:26:43.641338 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:43.641301 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39be085b-e445-416b-93c6-52de82dc6a83-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-njbt4\" (UID: \"39be085b-e445-416b-93c6-52de82dc6a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4" Oct 30 01:26:43.641461 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:43.641436 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39be085b-e445-416b-93c6-52de82dc6a83-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-njbt4\" (UID: \"39be085b-e445-416b-93c6-52de82dc6a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4" Oct 30 01:26:43.641739 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:43.641716 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39be085b-e445-416b-93c6-52de82dc6a83-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-njbt4\" (UID: \"39be085b-e445-416b-93c6-52de82dc6a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4" Oct 30 01:26:43.643235 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:43.643212 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39be085b-e445-416b-93c6-52de82dc6a83-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-njbt4\" (UID: \"39be085b-e445-416b-93c6-52de82dc6a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4" Oct 30 01:26:43.643499 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:43.643482 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39be085b-e445-416b-93c6-52de82dc6a83-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-njbt4\" (UID: \"39be085b-e445-416b-93c6-52de82dc6a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4" Oct 30 01:26:43.660725 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:43.660654 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39be085b-e445-416b-93c6-52de82dc6a83-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-njbt4\" (UID: \"39be085b-e445-416b-93c6-52de82dc6a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4" Oct 30 01:26:43.660866 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:43.660842 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ztpr\" (UniqueName: \"kubernetes.io/projected/39be085b-e445-416b-93c6-52de82dc6a83-kube-api-access-2ztpr\") pod \"maas-default-gateway-openshift-default-5bc884dd87-njbt4\" (UID: \"39be085b-e445-416b-93c6-52de82dc6a83\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4" Oct 30 01:26:43.714402 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:43.714364 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4" Oct 30 01:26:44.029935 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:44.029848 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4"] Oct 30 01:26:44.034674 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:26:44.034647 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39be085b_e445_416b_93c6_52de82dc6a83.slice/crio-ddf38bb0bc42a40e57e9b58ef54a87fe14b0ce7bccf65f7872bf6f9ed52015ad WatchSource:0}: Error finding container ddf38bb0bc42a40e57e9b58ef54a87fe14b0ce7bccf65f7872bf6f9ed52015ad: Status 404 returned error can't find the container with id ddf38bb0bc42a40e57e9b58ef54a87fe14b0ce7bccf65f7872bf6f9ed52015ad Oct 30 01:26:44.037683 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:44.037652 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:26:44.037751 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:44.037717 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:26:44.037783 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:44.037749 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:26:44.581779 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:44.581744 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4" event={"ID":"39be085b-e445-416b-93c6-52de82dc6a83","Type":"ContainerStarted","Data":"4608e02e79850ed8a57ec28dbe23da1936d60464caab397aaf2d771cd5710af3"} Oct 30 01:26:44.582193 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:44.581793 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4" event={"ID":"39be085b-e445-416b-93c6-52de82dc6a83","Type":"ContainerStarted","Data":"ddf38bb0bc42a40e57e9b58ef54a87fe14b0ce7bccf65f7872bf6f9ed52015ad"} Oct 30 01:26:44.666467 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:44.666419 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4" podStartSLOduration=1.66640532 podStartE2EDuration="1.66640532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:26:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:26:44.653512515 +0000 UTC m=+528.231799852" watchObservedRunningTime="2026-10-30 01:26:44.66640532 +0000 UTC m=+528.244692656" Oct 30 01:26:44.725565 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:44.721642 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4" Oct 30 01:26:45.726364 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:45.726323 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-njbt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 01:26:45.727029 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:45.726992 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4" podUID="39be085b-e445-416b-93c6-52de82dc6a83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 01:26:46.720131 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:46.720092 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-njbt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 01:26:46.720510 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:46.720409 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4" podUID="39be085b-e445-416b-93c6-52de82dc6a83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 01:26:47.724041 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:47.723995 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-njbt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 01:26:47.724676 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:47.724641 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4" podUID="39be085b-e445-416b-93c6-52de82dc6a83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 01:26:48.727553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:48.722961 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-njbt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 01:26:48.727553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:48.723025 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4" podUID="39be085b-e445-416b-93c6-52de82dc6a83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 01:26:49.721973 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:49.721929 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-njbt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 01:26:49.722269 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:49.722240 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4" podUID="39be085b-e445-416b-93c6-52de82dc6a83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 01:26:50.714737 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:50.714700 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-njbt4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 01:26:50.715217 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:50.714750 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4" podUID="39be085b-e445-416b-93c6-52de82dc6a83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 01:26:51.496758 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:26:51.496634 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:26:52.237064 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:52.237030 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4" Oct 30 01:26:52.237554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:52.237419 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4" Oct 30 01:26:52.238302 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:52.238275 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4" Oct 30 01:26:58.449115 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:58.449079 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk"] Oct 30 01:26:58.453462 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:58.453437 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk" Oct 30 01:26:58.487083 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:58.487050 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk"] Oct 30 01:26:58.585617 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:58.585570 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd15071b-edcd-4d7f-8309-c02a3281dcb5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qb7jk\" (UID: \"cd15071b-edcd-4d7f-8309-c02a3281dcb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk" Oct 30 01:26:58.585617 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:58.585620 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27md5\" (UniqueName: \"kubernetes.io/projected/cd15071b-edcd-4d7f-8309-c02a3281dcb5-kube-api-access-27md5\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qb7jk\" (UID: \"cd15071b-edcd-4d7f-8309-c02a3281dcb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk" Oct 30 01:26:58.585837 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:58.585727 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd15071b-edcd-4d7f-8309-c02a3281dcb5-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qb7jk\" (UID: \"cd15071b-edcd-4d7f-8309-c02a3281dcb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk" Oct 30 01:26:58.585837 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:58.585787 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd15071b-edcd-4d7f-8309-c02a3281dcb5-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qb7jk\" (UID: \"cd15071b-edcd-4d7f-8309-c02a3281dcb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk" Oct 30 01:26:58.585837 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:58.585823 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd15071b-edcd-4d7f-8309-c02a3281dcb5-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qb7jk\" (UID: \"cd15071b-edcd-4d7f-8309-c02a3281dcb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk" Oct 30 01:26:58.585990 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:58.585861 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd15071b-edcd-4d7f-8309-c02a3281dcb5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qb7jk\" (UID: \"cd15071b-edcd-4d7f-8309-c02a3281dcb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk" Oct 30 01:26:58.585990 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:58.585962 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd15071b-edcd-4d7f-8309-c02a3281dcb5-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qb7jk\" (UID: \"cd15071b-edcd-4d7f-8309-c02a3281dcb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk" Oct 30 01:26:58.586084 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:58.586019 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd15071b-edcd-4d7f-8309-c02a3281dcb5-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qb7jk\" (UID: \"cd15071b-edcd-4d7f-8309-c02a3281dcb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk" Oct 30 01:26:58.586084 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:58.586070 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd15071b-edcd-4d7f-8309-c02a3281dcb5-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qb7jk\" (UID: \"cd15071b-edcd-4d7f-8309-c02a3281dcb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk" Oct 30 01:26:58.687247 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:58.687208 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd15071b-edcd-4d7f-8309-c02a3281dcb5-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qb7jk\" (UID: \"cd15071b-edcd-4d7f-8309-c02a3281dcb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk" Oct 30 01:26:58.687461 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:58.687264 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd15071b-edcd-4d7f-8309-c02a3281dcb5-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qb7jk\" (UID: \"cd15071b-edcd-4d7f-8309-c02a3281dcb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk" Oct 30 01:26:58.687461 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:58.687319 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd15071b-edcd-4d7f-8309-c02a3281dcb5-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qb7jk\" (UID: \"cd15071b-edcd-4d7f-8309-c02a3281dcb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk" Oct 30 01:26:58.687461 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:58.687352 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd15071b-edcd-4d7f-8309-c02a3281dcb5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qb7jk\" (UID: \"cd15071b-edcd-4d7f-8309-c02a3281dcb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk" Oct 30 01:26:58.687461 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:58.687372 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27md5\" (UniqueName: \"kubernetes.io/projected/cd15071b-edcd-4d7f-8309-c02a3281dcb5-kube-api-access-27md5\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qb7jk\" (UID: \"cd15071b-edcd-4d7f-8309-c02a3281dcb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk" Oct 30 01:26:58.687461 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:58.687434 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd15071b-edcd-4d7f-8309-c02a3281dcb5-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qb7jk\" (UID: \"cd15071b-edcd-4d7f-8309-c02a3281dcb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk" Oct 30 01:26:58.687787 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:58.687557 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd15071b-edcd-4d7f-8309-c02a3281dcb5-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qb7jk\" (UID: \"cd15071b-edcd-4d7f-8309-c02a3281dcb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk" Oct 30 01:26:58.687787 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:58.687584 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd15071b-edcd-4d7f-8309-c02a3281dcb5-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qb7jk\" (UID: \"cd15071b-edcd-4d7f-8309-c02a3281dcb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk" Oct 30 01:26:58.687787 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:58.687608 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd15071b-edcd-4d7f-8309-c02a3281dcb5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qb7jk\" (UID: \"cd15071b-edcd-4d7f-8309-c02a3281dcb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk" Oct 30 01:26:58.687949 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:58.687804 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd15071b-edcd-4d7f-8309-c02a3281dcb5-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qb7jk\" (UID: \"cd15071b-edcd-4d7f-8309-c02a3281dcb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk" Oct 30 01:26:58.687949 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:58.687858 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd15071b-edcd-4d7f-8309-c02a3281dcb5-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qb7jk\" (UID: \"cd15071b-edcd-4d7f-8309-c02a3281dcb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk" Oct 30 01:26:58.688056 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:58.687990 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd15071b-edcd-4d7f-8309-c02a3281dcb5-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qb7jk\" (UID: \"cd15071b-edcd-4d7f-8309-c02a3281dcb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk" Oct 30 01:26:58.688225 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:58.688204 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd15071b-edcd-4d7f-8309-c02a3281dcb5-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qb7jk\" (UID: \"cd15071b-edcd-4d7f-8309-c02a3281dcb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk" Oct 30 01:26:58.688305 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:58.688288 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd15071b-edcd-4d7f-8309-c02a3281dcb5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qb7jk\" (UID: \"cd15071b-edcd-4d7f-8309-c02a3281dcb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk" Oct 30 01:26:58.689912 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:58.689890 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd15071b-edcd-4d7f-8309-c02a3281dcb5-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qb7jk\" (UID: \"cd15071b-edcd-4d7f-8309-c02a3281dcb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk" Oct 30 01:26:58.690019 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:58.690001 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd15071b-edcd-4d7f-8309-c02a3281dcb5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qb7jk\" (UID: \"cd15071b-edcd-4d7f-8309-c02a3281dcb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk" Oct 30 01:26:58.707642 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:58.707579 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd15071b-edcd-4d7f-8309-c02a3281dcb5-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qb7jk\" (UID: \"cd15071b-edcd-4d7f-8309-c02a3281dcb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk" Oct 30 01:26:58.707759 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:58.707696 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27md5\" (UniqueName: \"kubernetes.io/projected/cd15071b-edcd-4d7f-8309-c02a3281dcb5-kube-api-access-27md5\") pod \"maas-default-gateway-openshift-default-5bc884dd87-qb7jk\" (UID: \"cd15071b-edcd-4d7f-8309-c02a3281dcb5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk" Oct 30 01:26:58.767452 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:58.767419 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk" Oct 30 01:26:59.080929 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:59.080898 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk"] Oct 30 01:26:59.084023 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:26:59.083991 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd15071b_edcd_4d7f_8309_c02a3281dcb5.slice/crio-5675e86a59121cafd0a55c731a9ec79dc20398c640fd0436b8f6febb499b9535 WatchSource:0}: Error finding container 5675e86a59121cafd0a55c731a9ec79dc20398c640fd0436b8f6febb499b9535: Status 404 returned error can't find the container with id 5675e86a59121cafd0a55c731a9ec79dc20398c640fd0436b8f6febb499b9535 Oct 30 01:26:59.086211 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:59.086182 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:26:59.086280 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:59.086248 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:26:59.086322 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:59.086279 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:26:59.647601 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:59.647561 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk" event={"ID":"cd15071b-edcd-4d7f-8309-c02a3281dcb5","Type":"ContainerStarted","Data":"15c8a196d36428f6ab70ca0f10fb46afaca0f513cefe290c916400a7ee054b5b"} Oct 30 01:26:59.648110 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:59.647612 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk" event={"ID":"cd15071b-edcd-4d7f-8309-c02a3281dcb5","Type":"ContainerStarted","Data":"5675e86a59121cafd0a55c731a9ec79dc20398c640fd0436b8f6febb499b9535"} Oct 30 01:26:59.690078 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:59.690017 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk" podStartSLOduration=1.689999665 podStartE2EDuration="1.689999665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:26:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:26:59.688416936 +0000 UTC m=+543.266704282" watchObservedRunningTime="2026-10-30 01:26:59.689999665 +0000 UTC m=+543.268287003" Oct 30 01:26:59.767869 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:26:59.767826 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk" Oct 30 01:27:00.769135 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:00.769102 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qb7jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 01:27:00.769579 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:00.769151 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk" podUID="cd15071b-edcd-4d7f-8309-c02a3281dcb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 01:27:01.767940 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:01.767905 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qb7jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 01:27:01.768119 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:01.767967 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk" podUID="cd15071b-edcd-4d7f-8309-c02a3281dcb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 01:27:02.768492 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:02.768453 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qb7jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 01:27:02.768970 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:02.768517 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk" podUID="cd15071b-edcd-4d7f-8309-c02a3281dcb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 01:27:03.768900 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:03.768856 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qb7jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 01:27:03.769319 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:03.768919 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk" podUID="cd15071b-edcd-4d7f-8309-c02a3281dcb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 01:27:04.768246 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:04.768209 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qb7jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 01:27:04.768429 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:04.768279 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk" podUID="cd15071b-edcd-4d7f-8309-c02a3281dcb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 01:27:05.768196 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:05.768160 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qb7jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 01:27:05.768628 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:05.768216 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk" podUID="cd15071b-edcd-4d7f-8309-c02a3281dcb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 01:27:07.315149 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:07.315116 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk" Oct 30 01:27:07.315769 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:07.315747 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk" Oct 30 01:27:07.316476 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:07.316458 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk" Oct 30 01:27:18.248468 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.248393 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p"] Oct 30 01:27:18.251810 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.251794 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" Oct 30 01:27:18.282470 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.282446 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-5b724cd8-openshift-default-dockercfg-thnl5\"" Oct 30 01:27:18.308272 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.308248 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p"] Oct 30 01:27:18.400834 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.400800 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-istio-envoy\") pod \"e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p\" (UID: \"ec361aac-99e4-4f3c-9c0a-33411f8f74e3\") " pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" Oct 30 01:27:18.401021 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.400843 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-credential-socket\") pod \"e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p\" (UID: \"ec361aac-99e4-4f3c-9c0a-33411f8f74e3\") " pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" Oct 30 01:27:18.401021 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.400864 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-workload-socket\") pod \"e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p\" (UID: \"ec361aac-99e4-4f3c-9c0a-33411f8f74e3\") " pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" Oct 30 01:27:18.401021 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.400938 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkffw\" (UniqueName: \"kubernetes.io/projected/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-kube-api-access-gkffw\") pod \"e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p\" (UID: \"ec361aac-99e4-4f3c-9c0a-33411f8f74e3\") " pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" Oct 30 01:27:18.401134 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.401040 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-workload-certs\") pod \"e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p\" (UID: \"ec361aac-99e4-4f3c-9c0a-33411f8f74e3\") " pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" Oct 30 01:27:18.401134 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.401102 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-istio-token\") pod \"e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p\" (UID: \"ec361aac-99e4-4f3c-9c0a-33411f8f74e3\") " pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" Oct 30 01:27:18.401134 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.401125 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-istio-podinfo\") pod \"e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p\" (UID: \"ec361aac-99e4-4f3c-9c0a-33411f8f74e3\") " pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" Oct 30 01:27:18.401274 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.401153 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-istiod-ca-cert\") pod \"e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p\" (UID: \"ec361aac-99e4-4f3c-9c0a-33411f8f74e3\") " pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" Oct 30 01:27:18.401341 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.401319 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-istio-data\") pod \"e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p\" (UID: \"ec361aac-99e4-4f3c-9c0a-33411f8f74e3\") " pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" Oct 30 01:27:18.451955 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.451924 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj"] Oct 30 01:27:18.455353 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.455333 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" Oct 30 01:27:18.475999 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.475479 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f29772b9-openshift-default-dockercfg-rptjc\"" Oct 30 01:27:18.477735 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.477710 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj"] Oct 30 01:27:18.502500 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.502436 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-istio-token\") pod \"e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p\" (UID: \"ec361aac-99e4-4f3c-9c0a-33411f8f74e3\") " pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" Oct 30 01:27:18.502500 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.502475 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-istio-podinfo\") pod \"e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p\" (UID: \"ec361aac-99e4-4f3c-9c0a-33411f8f74e3\") " pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" Oct 30 01:27:18.502727 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.502594 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-istiod-ca-cert\") pod \"e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p\" (UID: \"ec361aac-99e4-4f3c-9c0a-33411f8f74e3\") " pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" Oct 30 01:27:18.502799 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.502780 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-istio-data\") pod \"e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p\" (UID: \"ec361aac-99e4-4f3c-9c0a-33411f8f74e3\") " pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" Oct 30 01:27:18.502872 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.502857 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-istio-envoy\") pod \"e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p\" (UID: \"ec361aac-99e4-4f3c-9c0a-33411f8f74e3\") " pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" Oct 30 01:27:18.503004 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.502980 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-credential-socket\") pod \"e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p\" (UID: \"ec361aac-99e4-4f3c-9c0a-33411f8f74e3\") " pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" Oct 30 01:27:18.503144 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.503011 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-workload-socket\") pod \"e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p\" (UID: \"ec361aac-99e4-4f3c-9c0a-33411f8f74e3\") " pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" Oct 30 01:27:18.503144 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.503040 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkffw\" (UniqueName: \"kubernetes.io/projected/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-kube-api-access-gkffw\") pod \"e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p\" (UID: \"ec361aac-99e4-4f3c-9c0a-33411f8f74e3\") " pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" Oct 30 01:27:18.503144 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.503067 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-workload-certs\") pod \"e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p\" (UID: \"ec361aac-99e4-4f3c-9c0a-33411f8f74e3\") " pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" Oct 30 01:27:18.503326 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.503275 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-istio-data\") pod \"e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p\" (UID: \"ec361aac-99e4-4f3c-9c0a-33411f8f74e3\") " pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" Oct 30 01:27:18.503393 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.503344 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-istiod-ca-cert\") pod \"e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p\" (UID: \"ec361aac-99e4-4f3c-9c0a-33411f8f74e3\") " pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" Oct 30 01:27:18.503393 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.503364 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-workload-certs\") pod \"e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p\" (UID: \"ec361aac-99e4-4f3c-9c0a-33411f8f74e3\") " pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" Oct 30 01:27:18.503506 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.503463 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-workload-socket\") pod \"e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p\" (UID: \"ec361aac-99e4-4f3c-9c0a-33411f8f74e3\") " pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" Oct 30 01:27:18.503663 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.503639 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-credential-socket\") pod \"e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p\" (UID: \"ec361aac-99e4-4f3c-9c0a-33411f8f74e3\") " pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" Oct 30 01:27:18.505292 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.505272 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-istio-podinfo\") pod \"e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p\" (UID: \"ec361aac-99e4-4f3c-9c0a-33411f8f74e3\") " pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" Oct 30 01:27:18.505455 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.505440 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-istio-envoy\") pod \"e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p\" (UID: \"ec361aac-99e4-4f3c-9c0a-33411f8f74e3\") " pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" Oct 30 01:27:18.522247 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.522221 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-istio-token\") pod \"e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p\" (UID: \"ec361aac-99e4-4f3c-9c0a-33411f8f74e3\") " pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" Oct 30 01:27:18.522591 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.522574 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkffw\" (UniqueName: \"kubernetes.io/projected/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-kube-api-access-gkffw\") pod \"e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p\" (UID: \"ec361aac-99e4-4f3c-9c0a-33411f8f74e3\") " pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" Oct 30 01:27:18.564198 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.564166 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" Oct 30 01:27:18.604101 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.603988 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03ccf6ef-c34a-4847-9791-1050db4d14d5-workload-certs\") pod \"e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj\" (UID: \"03ccf6ef-c34a-4847-9791-1050db4d14d5\") " pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" Oct 30 01:27:18.604101 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.604056 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcv7g\" (UniqueName: \"kubernetes.io/projected/03ccf6ef-c34a-4847-9791-1050db4d14d5-kube-api-access-dcv7g\") pod \"e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj\" (UID: \"03ccf6ef-c34a-4847-9791-1050db4d14d5\") " pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" Oct 30 01:27:18.604101 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.604088 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03ccf6ef-c34a-4847-9791-1050db4d14d5-istio-token\") pod \"e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj\" (UID: \"03ccf6ef-c34a-4847-9791-1050db4d14d5\") " pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" Oct 30 01:27:18.604365 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.604115 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ccf6ef-c34a-4847-9791-1050db4d14d5-credential-socket\") pod \"e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj\" (UID: \"03ccf6ef-c34a-4847-9791-1050db4d14d5\") " pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" Oct 30 01:27:18.604365 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.604190 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03ccf6ef-c34a-4847-9791-1050db4d14d5-istio-envoy\") pod \"e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj\" (UID: \"03ccf6ef-c34a-4847-9791-1050db4d14d5\") " pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" Oct 30 01:27:18.604365 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.604299 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03ccf6ef-c34a-4847-9791-1050db4d14d5-istiod-ca-cert\") pod \"e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj\" (UID: \"03ccf6ef-c34a-4847-9791-1050db4d14d5\") " pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" Oct 30 01:27:18.604517 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.604368 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03ccf6ef-c34a-4847-9791-1050db4d14d5-istio-data\") pod \"e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj\" (UID: \"03ccf6ef-c34a-4847-9791-1050db4d14d5\") " pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" Oct 30 01:27:18.604618 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.604574 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03ccf6ef-c34a-4847-9791-1050db4d14d5-istio-podinfo\") pod \"e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj\" (UID: \"03ccf6ef-c34a-4847-9791-1050db4d14d5\") " pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" Oct 30 01:27:18.604677 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.604651 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ccf6ef-c34a-4847-9791-1050db4d14d5-workload-socket\") pod \"e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj\" (UID: \"03ccf6ef-c34a-4847-9791-1050db4d14d5\") " pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" Oct 30 01:27:18.705439 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.705403 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03ccf6ef-c34a-4847-9791-1050db4d14d5-workload-certs\") pod \"e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj\" (UID: \"03ccf6ef-c34a-4847-9791-1050db4d14d5\") " pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" Oct 30 01:27:18.705634 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.705452 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcv7g\" (UniqueName: \"kubernetes.io/projected/03ccf6ef-c34a-4847-9791-1050db4d14d5-kube-api-access-dcv7g\") pod \"e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj\" (UID: \"03ccf6ef-c34a-4847-9791-1050db4d14d5\") " pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" Oct 30 01:27:18.705634 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.705480 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03ccf6ef-c34a-4847-9791-1050db4d14d5-istio-token\") pod \"e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj\" (UID: \"03ccf6ef-c34a-4847-9791-1050db4d14d5\") " pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" Oct 30 01:27:18.705634 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.705507 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ccf6ef-c34a-4847-9791-1050db4d14d5-credential-socket\") pod \"e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj\" (UID: \"03ccf6ef-c34a-4847-9791-1050db4d14d5\") " pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" Oct 30 01:27:18.705634 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.705559 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03ccf6ef-c34a-4847-9791-1050db4d14d5-istio-envoy\") pod \"e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj\" (UID: \"03ccf6ef-c34a-4847-9791-1050db4d14d5\") " pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" Oct 30 01:27:18.705634 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.705603 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03ccf6ef-c34a-4847-9791-1050db4d14d5-istiod-ca-cert\") pod \"e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj\" (UID: \"03ccf6ef-c34a-4847-9791-1050db4d14d5\") " pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" Oct 30 01:27:18.705923 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.705657 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03ccf6ef-c34a-4847-9791-1050db4d14d5-istio-data\") pod \"e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj\" (UID: \"03ccf6ef-c34a-4847-9791-1050db4d14d5\") " pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" Oct 30 01:27:18.705923 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.705717 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03ccf6ef-c34a-4847-9791-1050db4d14d5-istio-podinfo\") pod \"e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj\" (UID: \"03ccf6ef-c34a-4847-9791-1050db4d14d5\") " pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" Oct 30 01:27:18.705923 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.705765 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ccf6ef-c34a-4847-9791-1050db4d14d5-workload-socket\") pod \"e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj\" (UID: \"03ccf6ef-c34a-4847-9791-1050db4d14d5\") " pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" Oct 30 01:27:18.705923 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.705881 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03ccf6ef-c34a-4847-9791-1050db4d14d5-workload-certs\") pod \"e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj\" (UID: \"03ccf6ef-c34a-4847-9791-1050db4d14d5\") " pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" Oct 30 01:27:18.706135 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.705956 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ccf6ef-c34a-4847-9791-1050db4d14d5-credential-socket\") pod \"e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj\" (UID: \"03ccf6ef-c34a-4847-9791-1050db4d14d5\") " pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" Oct 30 01:27:18.706135 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.706104 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ccf6ef-c34a-4847-9791-1050db4d14d5-workload-socket\") pod \"e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj\" (UID: \"03ccf6ef-c34a-4847-9791-1050db4d14d5\") " pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" Oct 30 01:27:18.706243 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.706182 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03ccf6ef-c34a-4847-9791-1050db4d14d5-istio-data\") pod \"e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj\" (UID: \"03ccf6ef-c34a-4847-9791-1050db4d14d5\") " pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" Oct 30 01:27:18.706897 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.706864 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03ccf6ef-c34a-4847-9791-1050db4d14d5-istiod-ca-cert\") pod \"e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj\" (UID: \"03ccf6ef-c34a-4847-9791-1050db4d14d5\") " pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" Oct 30 01:27:18.708234 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.708205 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03ccf6ef-c34a-4847-9791-1050db4d14d5-istio-envoy\") pod \"e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj\" (UID: \"03ccf6ef-c34a-4847-9791-1050db4d14d5\") " pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" Oct 30 01:27:18.708944 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.708923 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03ccf6ef-c34a-4847-9791-1050db4d14d5-istio-podinfo\") pod \"e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj\" (UID: \"03ccf6ef-c34a-4847-9791-1050db4d14d5\") " pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" Oct 30 01:27:18.731373 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.731348 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03ccf6ef-c34a-4847-9791-1050db4d14d5-istio-token\") pod \"e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj\" (UID: \"03ccf6ef-c34a-4847-9791-1050db4d14d5\") " pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" Oct 30 01:27:18.731495 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.731483 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcv7g\" (UniqueName: \"kubernetes.io/projected/03ccf6ef-c34a-4847-9791-1050db4d14d5-kube-api-access-dcv7g\") pod \"e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj\" (UID: \"03ccf6ef-c34a-4847-9791-1050db4d14d5\") " pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" Oct 30 01:27:18.772827 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.772761 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" Oct 30 01:27:18.882475 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.882445 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p"] Oct 30 01:27:18.884317 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:27:18.884286 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec361aac_99e4_4f3c_9c0a_33411f8f74e3.slice/crio-3111975d96296cc52f86a33b26521b7824a3485bf80ad7b7f21434fd3452829b WatchSource:0}: Error finding container 3111975d96296cc52f86a33b26521b7824a3485bf80ad7b7f21434fd3452829b: Status 404 returned error can't find the container with id 3111975d96296cc52f86a33b26521b7824a3485bf80ad7b7f21434fd3452829b Oct 30 01:27:18.889488 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.889455 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:27:18.889586 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.889520 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:27:18.889586 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:18.889566 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:27:19.074358 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:19.074328 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj"] Oct 30 01:27:19.078132 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:27:19.078104 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03ccf6ef_c34a_4847_9791_1050db4d14d5.slice/crio-be4f44d419c95bb440d772cb8b27672b9d48c0a0ffe6776992893bf2e5e73ba0 WatchSource:0}: Error finding container be4f44d419c95bb440d772cb8b27672b9d48c0a0ffe6776992893bf2e5e73ba0: Status 404 returned error can't find the container with id be4f44d419c95bb440d772cb8b27672b9d48c0a0ffe6776992893bf2e5e73ba0 Oct 30 01:27:19.080506 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:19.080476 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:27:19.080590 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:19.080560 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:27:19.080645 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:19.080594 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:27:19.736426 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:19.736384 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" event={"ID":"03ccf6ef-c34a-4847-9791-1050db4d14d5","Type":"ContainerStarted","Data":"f2ac4d800448acebbc51e1f8e6ab4063ade4400929aeecef50b9a112deaa126c"} Oct 30 01:27:19.736886 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:19.736452 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" event={"ID":"03ccf6ef-c34a-4847-9791-1050db4d14d5","Type":"ContainerStarted","Data":"be4f44d419c95bb440d772cb8b27672b9d48c0a0ffe6776992893bf2e5e73ba0"} Oct 30 01:27:19.737906 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:19.737882 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" event={"ID":"ec361aac-99e4-4f3c-9c0a-33411f8f74e3","Type":"ContainerStarted","Data":"24ede9319bbff330d24a8f9e1c538c3dbfd69f8bf0c6c448f05c44837902ef50"} Oct 30 01:27:19.737961 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:19.737924 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" event={"ID":"ec361aac-99e4-4f3c-9c0a-33411f8f74e3","Type":"ContainerStarted","Data":"3111975d96296cc52f86a33b26521b7824a3485bf80ad7b7f21434fd3452829b"} Oct 30 01:27:19.773448 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:19.773422 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" Oct 30 01:27:19.782638 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:19.782598 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" podStartSLOduration=1.7825866449999999 podStartE2EDuration="1.782586645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:19.782143267 +0000 UTC m=+563.360430605" watchObservedRunningTime="2026-10-30 01:27:19.782586645 +0000 UTC m=+563.360873980" Oct 30 01:27:19.825607 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:19.825561 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" podStartSLOduration=1.825545486 podStartE2EDuration="1.825545486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:19.823698795 +0000 UTC m=+563.401986132" watchObservedRunningTime="2026-10-30 01:27:19.825545486 +0000 UTC m=+563.403832821" Oct 30 01:27:20.565216 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:20.565185 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" Oct 30 01:27:20.570408 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:20.570385 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" Oct 30 01:27:20.741970 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:20.741927 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" Oct 30 01:27:20.742755 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:20.742734 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" Oct 30 01:27:20.778065 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:20.778044 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" Oct 30 01:27:21.494277 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:27:21.494244 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:27:21.745800 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:21.745722 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" Oct 30 01:27:21.746705 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:21.746683 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" Oct 30 01:27:23.178184 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:23.178152 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d9801220-6c74684cd7-4x5z4"] Oct 30 01:27:23.181751 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:23.181736 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d9801220-6c74684cd7-4x5z4" Oct 30 01:27:23.199337 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:23.199311 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d9801220-6c74684cd7-4x5z4"] Oct 30 01:27:23.210181 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:23.210157 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-d9801220\"" Oct 30 01:27:23.210286 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:23.210217 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-d9801220-dockercfg-dsxl8\"" Oct 30 01:27:23.254295 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:23.254239 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2xzf\" (UniqueName: \"kubernetes.io/projected/f0d2d13d-4eb6-4a8a-8604-d893dbfe03df-kube-api-access-s2xzf\") pod \"payload-pre-processing-e2e-migrate-d9801220-6c74684cd7-4x5z4\" (UID: \"f0d2d13d-4eb6-4a8a-8604-d893dbfe03df\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d9801220-6c74684cd7-4x5z4" Oct 30 01:27:23.254570 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:23.254324 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0d2d13d-4eb6-4a8a-8604-d893dbfe03df-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d9801220-6c74684cd7-4x5z4\" (UID: \"f0d2d13d-4eb6-4a8a-8604-d893dbfe03df\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d9801220-6c74684cd7-4x5z4" Oct 30 01:27:23.281628 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:23.280407 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d9801220-7df68db76d-zrpzc"] Oct 30 01:27:23.287068 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:23.287040 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d9801220-7df68db76d-zrpzc" Oct 30 01:27:23.303245 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:23.303209 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d9801220-7df68db76d-zrpzc"] Oct 30 01:27:23.355768 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:23.355739 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0d2d13d-4eb6-4a8a-8604-d893dbfe03df-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d9801220-6c74684cd7-4x5z4\" (UID: \"f0d2d13d-4eb6-4a8a-8604-d893dbfe03df\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d9801220-6c74684cd7-4x5z4" Oct 30 01:27:23.355921 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:23.355781 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfk7j\" (UniqueName: \"kubernetes.io/projected/2e4c7a82-6d00-4b93-bad8-f20425f04d40-kube-api-access-pfk7j\") pod \"payload-processing-e2e-migrate-d9801220-7df68db76d-zrpzc\" (UID: \"2e4c7a82-6d00-4b93-bad8-f20425f04d40\") " pod="openshift-ingress/payload-processing-e2e-migrate-d9801220-7df68db76d-zrpzc" Oct 30 01:27:23.355921 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:23.355852 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e4c7a82-6d00-4b93-bad8-f20425f04d40-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d9801220-7df68db76d-zrpzc\" (UID: \"2e4c7a82-6d00-4b93-bad8-f20425f04d40\") " pod="openshift-ingress/payload-processing-e2e-migrate-d9801220-7df68db76d-zrpzc" Oct 30 01:27:23.356008 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:23.355983 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2xzf\" (UniqueName: \"kubernetes.io/projected/f0d2d13d-4eb6-4a8a-8604-d893dbfe03df-kube-api-access-s2xzf\") pod \"payload-pre-processing-e2e-migrate-d9801220-6c74684cd7-4x5z4\" (UID: \"f0d2d13d-4eb6-4a8a-8604-d893dbfe03df\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d9801220-6c74684cd7-4x5z4" Oct 30 01:27:23.356471 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:23.356455 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0d2d13d-4eb6-4a8a-8604-d893dbfe03df-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d9801220-6c74684cd7-4x5z4\" (UID: \"f0d2d13d-4eb6-4a8a-8604-d893dbfe03df\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d9801220-6c74684cd7-4x5z4" Oct 30 01:27:23.375576 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:23.375553 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2xzf\" (UniqueName: \"kubernetes.io/projected/f0d2d13d-4eb6-4a8a-8604-d893dbfe03df-kube-api-access-s2xzf\") pod \"payload-pre-processing-e2e-migrate-d9801220-6c74684cd7-4x5z4\" (UID: \"f0d2d13d-4eb6-4a8a-8604-d893dbfe03df\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d9801220-6c74684cd7-4x5z4" Oct 30 01:27:23.456859 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:23.456789 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfk7j\" (UniqueName: \"kubernetes.io/projected/2e4c7a82-6d00-4b93-bad8-f20425f04d40-kube-api-access-pfk7j\") pod \"payload-processing-e2e-migrate-d9801220-7df68db76d-zrpzc\" (UID: \"2e4c7a82-6d00-4b93-bad8-f20425f04d40\") " pod="openshift-ingress/payload-processing-e2e-migrate-d9801220-7df68db76d-zrpzc" Oct 30 01:27:23.456995 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:23.456882 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e4c7a82-6d00-4b93-bad8-f20425f04d40-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d9801220-7df68db76d-zrpzc\" (UID: \"2e4c7a82-6d00-4b93-bad8-f20425f04d40\") " pod="openshift-ingress/payload-processing-e2e-migrate-d9801220-7df68db76d-zrpzc" Oct 30 01:27:23.457569 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:23.457548 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e4c7a82-6d00-4b93-bad8-f20425f04d40-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d9801220-7df68db76d-zrpzc\" (UID: \"2e4c7a82-6d00-4b93-bad8-f20425f04d40\") " pod="openshift-ingress/payload-processing-e2e-migrate-d9801220-7df68db76d-zrpzc" Oct 30 01:27:23.477896 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:23.477866 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfk7j\" (UniqueName: \"kubernetes.io/projected/2e4c7a82-6d00-4b93-bad8-f20425f04d40-kube-api-access-pfk7j\") pod \"payload-processing-e2e-migrate-d9801220-7df68db76d-zrpzc\" (UID: \"2e4c7a82-6d00-4b93-bad8-f20425f04d40\") " pod="openshift-ingress/payload-processing-e2e-migrate-d9801220-7df68db76d-zrpzc" Oct 30 01:27:23.494758 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:23.494731 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d9801220-6c74684cd7-4x5z4" Oct 30 01:27:23.607463 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:23.607434 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d9801220-7df68db76d-zrpzc" Oct 30 01:27:23.799501 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:23.799475 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d9801220-6c74684cd7-4x5z4"] Oct 30 01:27:23.803010 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:27:23.802981 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0d2d13d_4eb6_4a8a_8604_d893dbfe03df.slice/crio-572ea634f837bee0052600cb5d85abcbf9dc1d7534bded1fd2e3f52cbaa5eb96 WatchSource:0}: Error finding container 572ea634f837bee0052600cb5d85abcbf9dc1d7534bded1fd2e3f52cbaa5eb96: Status 404 returned error can't find the container with id 572ea634f837bee0052600cb5d85abcbf9dc1d7534bded1fd2e3f52cbaa5eb96 Oct 30 01:27:23.909339 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:23.909304 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d9801220-7df68db76d-zrpzc"] Oct 30 01:27:23.913290 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:27:23.913259 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e4c7a82_6d00_4b93_bad8_f20425f04d40.slice/crio-ad57a65e02c18f73a4a2ad83374a0f3a4bd512283e43d9611f2d0eb549e53433 WatchSource:0}: Error finding container ad57a65e02c18f73a4a2ad83374a0f3a4bd512283e43d9611f2d0eb549e53433: Status 404 returned error can't find the container with id ad57a65e02c18f73a4a2ad83374a0f3a4bd512283e43d9611f2d0eb549e53433 Oct 30 01:27:24.686389 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:24.686357 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d9801220-f94447d96-7sv8l"] Oct 30 01:27:24.690104 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:24.690082 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9801220-f94447d96-7sv8l" Oct 30 01:27:24.705986 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:24.705964 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-d9801220-serving-cert\"" Oct 30 01:27:24.723489 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:24.723458 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d9801220-f94447d96-7sv8l"] Oct 30 01:27:24.760461 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:24.760418 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d9801220-7df68db76d-zrpzc" event={"ID":"2e4c7a82-6d00-4b93-bad8-f20425f04d40","Type":"ContainerStarted","Data":"201637b89e0f197a1b6748842ac49b767c952d3282255117385f56497b9e4ad9"} Oct 30 01:27:24.760683 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:24.760486 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d9801220-7df68db76d-zrpzc" event={"ID":"2e4c7a82-6d00-4b93-bad8-f20425f04d40","Type":"ContainerStarted","Data":"ad57a65e02c18f73a4a2ad83374a0f3a4bd512283e43d9611f2d0eb549e53433"} Oct 30 01:27:24.760683 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:24.760519 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-d9801220-7df68db76d-zrpzc" Oct 30 01:27:24.761945 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:24.761907 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d9801220-6c74684cd7-4x5z4" event={"ID":"f0d2d13d-4eb6-4a8a-8604-d893dbfe03df","Type":"ContainerStarted","Data":"6485652b905165fc5ef851f8543d2cecc7dadb640a8b324be248753bab75de59"} Oct 30 01:27:24.762077 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:24.761959 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d9801220-6c74684cd7-4x5z4" Oct 30 01:27:24.762077 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:24.761985 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d9801220-6c74684cd7-4x5z4" event={"ID":"f0d2d13d-4eb6-4a8a-8604-d893dbfe03df","Type":"ContainerStarted","Data":"572ea634f837bee0052600cb5d85abcbf9dc1d7534bded1fd2e3f52cbaa5eb96"} Oct 30 01:27:24.770825 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:24.770796 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b219867f-db12-4b12-918a-24d4151ea2b7-metrics-certs\") pod \"maas-api-e2e-migrate-d9801220-f94447d96-7sv8l\" (UID: \"b219867f-db12-4b12-918a-24d4151ea2b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9801220-f94447d96-7sv8l" Oct 30 01:27:24.770973 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:24.770954 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67cll\" (UniqueName: \"kubernetes.io/projected/b219867f-db12-4b12-918a-24d4151ea2b7-kube-api-access-67cll\") pod \"maas-api-e2e-migrate-d9801220-f94447d96-7sv8l\" (UID: \"b219867f-db12-4b12-918a-24d4151ea2b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9801220-f94447d96-7sv8l" Oct 30 01:27:24.771029 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:24.770994 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b219867f-db12-4b12-918a-24d4151ea2b7-openshift-service-ca\") pod \"maas-api-e2e-migrate-d9801220-f94447d96-7sv8l\" (UID: \"b219867f-db12-4b12-918a-24d4151ea2b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9801220-f94447d96-7sv8l" Oct 30 01:27:24.771029 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:24.771016 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b219867f-db12-4b12-918a-24d4151ea2b7-maas-api-tls\") pod \"maas-api-e2e-migrate-d9801220-f94447d96-7sv8l\" (UID: \"b219867f-db12-4b12-918a-24d4151ea2b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9801220-f94447d96-7sv8l" Oct 30 01:27:24.800282 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:24.800239 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-d9801220-7df68db76d-zrpzc" podStartSLOduration=1.8002276849999999 podStartE2EDuration="1.800227685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:24.799607268 +0000 UTC m=+568.377894604" watchObservedRunningTime="2026-10-30 01:27:24.800227685 +0000 UTC m=+568.378515020" Oct 30 01:27:24.834874 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:24.834843 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5b724cd8-6fd679fd8-phz5k"] Oct 30 01:27:24.838327 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:24.838312 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5b724cd8-6fd679fd8-phz5k" Oct 30 01:27:24.841129 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:24.841089 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d9801220-6c74684cd7-4x5z4" podStartSLOduration=1.841077239 podStartE2EDuration="1.841077239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:24.840211941 +0000 UTC m=+568.418499277" watchObservedRunningTime="2026-10-30 01:27:24.841077239 +0000 UTC m=+568.419364574" Oct 30 01:27:24.854785 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:24.854745 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-5b724cd8-dockercfg-wng9n\"" Oct 30 01:27:24.854897 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:24.854829 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-5b724cd8\"" Oct 30 01:27:24.856327 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:24.856300 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5b724cd8-6fd679fd8-phz5k"] Oct 30 01:27:24.872390 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:24.872363 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67cll\" (UniqueName: \"kubernetes.io/projected/b219867f-db12-4b12-918a-24d4151ea2b7-kube-api-access-67cll\") pod \"maas-api-e2e-migrate-d9801220-f94447d96-7sv8l\" (UID: \"b219867f-db12-4b12-918a-24d4151ea2b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9801220-f94447d96-7sv8l" Oct 30 01:27:24.872498 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:24.872440 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b219867f-db12-4b12-918a-24d4151ea2b7-openshift-service-ca\") pod \"maas-api-e2e-migrate-d9801220-f94447d96-7sv8l\" (UID: \"b219867f-db12-4b12-918a-24d4151ea2b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9801220-f94447d96-7sv8l" Oct 30 01:27:24.872498 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:24.872470 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b219867f-db12-4b12-918a-24d4151ea2b7-maas-api-tls\") pod \"maas-api-e2e-migrate-d9801220-f94447d96-7sv8l\" (UID: \"b219867f-db12-4b12-918a-24d4151ea2b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9801220-f94447d96-7sv8l" Oct 30 01:27:24.872651 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:24.872587 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b219867f-db12-4b12-918a-24d4151ea2b7-metrics-certs\") pod \"maas-api-e2e-migrate-d9801220-f94447d96-7sv8l\" (UID: \"b219867f-db12-4b12-918a-24d4151ea2b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9801220-f94447d96-7sv8l" Oct 30 01:27:24.873265 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:24.873235 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b219867f-db12-4b12-918a-24d4151ea2b7-openshift-service-ca\") pod \"maas-api-e2e-migrate-d9801220-f94447d96-7sv8l\" (UID: \"b219867f-db12-4b12-918a-24d4151ea2b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9801220-f94447d96-7sv8l" Oct 30 01:27:24.875116 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:24.875097 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b219867f-db12-4b12-918a-24d4151ea2b7-maas-api-tls\") pod \"maas-api-e2e-migrate-d9801220-f94447d96-7sv8l\" (UID: \"b219867f-db12-4b12-918a-24d4151ea2b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9801220-f94447d96-7sv8l" Oct 30 01:27:24.875201 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:24.875153 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b219867f-db12-4b12-918a-24d4151ea2b7-metrics-certs\") pod \"maas-api-e2e-migrate-d9801220-f94447d96-7sv8l\" (UID: \"b219867f-db12-4b12-918a-24d4151ea2b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9801220-f94447d96-7sv8l" Oct 30 01:27:24.894831 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:24.894806 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67cll\" (UniqueName: \"kubernetes.io/projected/b219867f-db12-4b12-918a-24d4151ea2b7-kube-api-access-67cll\") pod \"maas-api-e2e-migrate-d9801220-f94447d96-7sv8l\" (UID: \"b219867f-db12-4b12-918a-24d4151ea2b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9801220-f94447d96-7sv8l" Oct 30 01:27:24.977324 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:24.977026 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/313c4f85-855a-4e1a-b3d3-ae2d4e038bce-plugins-config-volume\") pod \"payload-processing-e2e-ait-5b724cd8-6fd679fd8-phz5k\" (UID: \"313c4f85-855a-4e1a-b3d3-ae2d4e038bce\") " pod="openshift-ingress/payload-processing-e2e-ait-5b724cd8-6fd679fd8-phz5k" Oct 30 01:27:24.977324 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:24.977097 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbnpz\" (UniqueName: \"kubernetes.io/projected/313c4f85-855a-4e1a-b3d3-ae2d4e038bce-kube-api-access-hbnpz\") pod \"payload-processing-e2e-ait-5b724cd8-6fd679fd8-phz5k\" (UID: \"313c4f85-855a-4e1a-b3d3-ae2d4e038bce\") " pod="openshift-ingress/payload-processing-e2e-ait-5b724cd8-6fd679fd8-phz5k" Oct 30 01:27:25.006854 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:25.006463 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9801220-f94447d96-7sv8l" Oct 30 01:27:25.078583 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:25.078552 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/313c4f85-855a-4e1a-b3d3-ae2d4e038bce-plugins-config-volume\") pod \"payload-processing-e2e-ait-5b724cd8-6fd679fd8-phz5k\" (UID: \"313c4f85-855a-4e1a-b3d3-ae2d4e038bce\") " pod="openshift-ingress/payload-processing-e2e-ait-5b724cd8-6fd679fd8-phz5k" Oct 30 01:27:25.078705 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:25.078604 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbnpz\" (UniqueName: \"kubernetes.io/projected/313c4f85-855a-4e1a-b3d3-ae2d4e038bce-kube-api-access-hbnpz\") pod \"payload-processing-e2e-ait-5b724cd8-6fd679fd8-phz5k\" (UID: \"313c4f85-855a-4e1a-b3d3-ae2d4e038bce\") " pod="openshift-ingress/payload-processing-e2e-ait-5b724cd8-6fd679fd8-phz5k" Oct 30 01:27:25.079391 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:25.079351 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/313c4f85-855a-4e1a-b3d3-ae2d4e038bce-plugins-config-volume\") pod \"payload-processing-e2e-ait-5b724cd8-6fd679fd8-phz5k\" (UID: \"313c4f85-855a-4e1a-b3d3-ae2d4e038bce\") " pod="openshift-ingress/payload-processing-e2e-ait-5b724cd8-6fd679fd8-phz5k" Oct 30 01:27:25.098712 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:25.098686 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbnpz\" (UniqueName: \"kubernetes.io/projected/313c4f85-855a-4e1a-b3d3-ae2d4e038bce-kube-api-access-hbnpz\") pod \"payload-processing-e2e-ait-5b724cd8-6fd679fd8-phz5k\" (UID: \"313c4f85-855a-4e1a-b3d3-ae2d4e038bce\") " pod="openshift-ingress/payload-processing-e2e-ait-5b724cd8-6fd679fd8-phz5k" Oct 30 01:27:25.150784 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:25.150742 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5b724cd8-6fd679fd8-phz5k" Oct 30 01:27:25.327107 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:25.327070 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d9801220-f94447d96-7sv8l"] Oct 30 01:27:25.330094 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:27:25.330071 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb219867f_db12_4b12_918a_24d4151ea2b7.slice/crio-345e6318123bf562bc6431ec92c4ee06a9ed6acc9999c463965fed01c09481f3 WatchSource:0}: Error finding container 345e6318123bf562bc6431ec92c4ee06a9ed6acc9999c463965fed01c09481f3: Status 404 returned error can't find the container with id 345e6318123bf562bc6431ec92c4ee06a9ed6acc9999c463965fed01c09481f3 Oct 30 01:27:25.451230 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:25.451192 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5b724cd8-6fd679fd8-phz5k"] Oct 30 01:27:25.454537 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:27:25.454496 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod313c4f85_855a_4e1a_b3d3_ae2d4e038bce.slice/crio-c51e1fc3adc970df84e0f8576829f9f4db26d09bb10654c8ac50f70a5fee80b6 WatchSource:0}: Error finding container c51e1fc3adc970df84e0f8576829f9f4db26d09bb10654c8ac50f70a5fee80b6: Status 404 returned error can't find the container with id c51e1fc3adc970df84e0f8576829f9f4db26d09bb10654c8ac50f70a5fee80b6 Oct 30 01:27:25.769387 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:25.769336 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5b724cd8-6fd679fd8-phz5k" event={"ID":"313c4f85-855a-4e1a-b3d3-ae2d4e038bce","Type":"ContainerStarted","Data":"f20a8cfd3a1a02aeaa5e646f9f88ec68c63a31dd5a1ca997bd5d918b0c7de595"} Oct 30 01:27:25.769821 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:25.769413 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5b724cd8-6fd679fd8-phz5k" event={"ID":"313c4f85-855a-4e1a-b3d3-ae2d4e038bce","Type":"ContainerStarted","Data":"c51e1fc3adc970df84e0f8576829f9f4db26d09bb10654c8ac50f70a5fee80b6"} Oct 30 01:27:25.769821 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:25.769447 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-5b724cd8-6fd679fd8-phz5k" Oct 30 01:27:25.771044 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:25.771017 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9801220-f94447d96-7sv8l" event={"ID":"b219867f-db12-4b12-918a-24d4151ea2b7","Type":"ContainerStarted","Data":"345e6318123bf562bc6431ec92c4ee06a9ed6acc9999c463965fed01c09481f3"} Oct 30 01:27:25.816809 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:25.816761 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-5b724cd8-6fd679fd8-phz5k" podStartSLOduration=1.816747217 podStartE2EDuration="1.816747217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:25.81510284 +0000 UTC m=+569.393390179" watchObservedRunningTime="2026-10-30 01:27:25.816747217 +0000 UTC m=+569.395034552" Oct 30 01:27:26.374706 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:26.374672 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f19ba569-5fb5787844-wnqfr"] Oct 30 01:27:26.378213 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:26.378194 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f19ba569-5fb5787844-wnqfr" Oct 30 01:27:26.397799 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:26.397775 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f19ba569-serving-cert\"" Oct 30 01:27:26.399961 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:26.399937 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f19ba569-5fb5787844-wnqfr"] Oct 30 01:27:26.495015 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:26.494977 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b1bcfa7-d075-4aa9-aca3-afd74f76a125-maas-api-tls\") pod \"maas-api-e2e-ait-f19ba569-5fb5787844-wnqfr\" (UID: \"7b1bcfa7-d075-4aa9-aca3-afd74f76a125\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f19ba569-5fb5787844-wnqfr" Oct 30 01:27:26.495158 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:26.495105 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b1bcfa7-d075-4aa9-aca3-afd74f76a125-openshift-service-ca\") pod \"maas-api-e2e-ait-f19ba569-5fb5787844-wnqfr\" (UID: \"7b1bcfa7-d075-4aa9-aca3-afd74f76a125\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f19ba569-5fb5787844-wnqfr" Oct 30 01:27:26.495158 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:26.495133 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b1bcfa7-d075-4aa9-aca3-afd74f76a125-metrics-certs\") pod \"maas-api-e2e-ait-f19ba569-5fb5787844-wnqfr\" (UID: \"7b1bcfa7-d075-4aa9-aca3-afd74f76a125\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f19ba569-5fb5787844-wnqfr" Oct 30 01:27:26.495261 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:26.495249 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqtcs\" (UniqueName: \"kubernetes.io/projected/7b1bcfa7-d075-4aa9-aca3-afd74f76a125-kube-api-access-qqtcs\") pod \"maas-api-e2e-ait-f19ba569-5fb5787844-wnqfr\" (UID: \"7b1bcfa7-d075-4aa9-aca3-afd74f76a125\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f19ba569-5fb5787844-wnqfr" Oct 30 01:27:26.596073 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:26.596034 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqtcs\" (UniqueName: \"kubernetes.io/projected/7b1bcfa7-d075-4aa9-aca3-afd74f76a125-kube-api-access-qqtcs\") pod \"maas-api-e2e-ait-f19ba569-5fb5787844-wnqfr\" (UID: \"7b1bcfa7-d075-4aa9-aca3-afd74f76a125\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f19ba569-5fb5787844-wnqfr" Oct 30 01:27:26.596073 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:26.596077 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b1bcfa7-d075-4aa9-aca3-afd74f76a125-maas-api-tls\") pod \"maas-api-e2e-ait-f19ba569-5fb5787844-wnqfr\" (UID: \"7b1bcfa7-d075-4aa9-aca3-afd74f76a125\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f19ba569-5fb5787844-wnqfr" Oct 30 01:27:26.596339 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:26.596163 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b1bcfa7-d075-4aa9-aca3-afd74f76a125-openshift-service-ca\") pod \"maas-api-e2e-ait-f19ba569-5fb5787844-wnqfr\" (UID: \"7b1bcfa7-d075-4aa9-aca3-afd74f76a125\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f19ba569-5fb5787844-wnqfr" Oct 30 01:27:26.596339 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:26.596190 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b1bcfa7-d075-4aa9-aca3-afd74f76a125-metrics-certs\") pod \"maas-api-e2e-ait-f19ba569-5fb5787844-wnqfr\" (UID: \"7b1bcfa7-d075-4aa9-aca3-afd74f76a125\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f19ba569-5fb5787844-wnqfr" Oct 30 01:27:26.596942 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:26.596909 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b1bcfa7-d075-4aa9-aca3-afd74f76a125-openshift-service-ca\") pod \"maas-api-e2e-ait-f19ba569-5fb5787844-wnqfr\" (UID: \"7b1bcfa7-d075-4aa9-aca3-afd74f76a125\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f19ba569-5fb5787844-wnqfr" Oct 30 01:27:26.598631 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:26.598604 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b1bcfa7-d075-4aa9-aca3-afd74f76a125-metrics-certs\") pod \"maas-api-e2e-ait-f19ba569-5fb5787844-wnqfr\" (UID: \"7b1bcfa7-d075-4aa9-aca3-afd74f76a125\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f19ba569-5fb5787844-wnqfr" Oct 30 01:27:26.598717 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:26.598695 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b1bcfa7-d075-4aa9-aca3-afd74f76a125-maas-api-tls\") pod \"maas-api-e2e-ait-f19ba569-5fb5787844-wnqfr\" (UID: \"7b1bcfa7-d075-4aa9-aca3-afd74f76a125\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f19ba569-5fb5787844-wnqfr" Oct 30 01:27:26.620266 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:26.620243 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqtcs\" (UniqueName: \"kubernetes.io/projected/7b1bcfa7-d075-4aa9-aca3-afd74f76a125-kube-api-access-qqtcs\") pod \"maas-api-e2e-ait-f19ba569-5fb5787844-wnqfr\" (UID: \"7b1bcfa7-d075-4aa9-aca3-afd74f76a125\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f19ba569-5fb5787844-wnqfr" Oct 30 01:27:26.691350 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:26.691266 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f19ba569-5fb5787844-wnqfr" Oct 30 01:27:26.779119 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:26.779078 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9801220-f94447d96-7sv8l" event={"ID":"b219867f-db12-4b12-918a-24d4151ea2b7","Type":"ContainerStarted","Data":"fbaca07c09731ac3325b0496786c2cc36c1120d56290a61b61f8788c0a7b39f7"} Oct 30 01:27:26.826561 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:26.826482 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9801220-f94447d96-7sv8l" podStartSLOduration=2.407450395 podStartE2EDuration="2.826462357s" totalImagesPullingTime="419.011962ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:26.826092983 +0000 UTC m=+570.404380332" watchObservedRunningTime="2026-10-30 01:27:26.826462357 +0000 UTC m=+570.404749693" Oct 30 01:27:27.014353 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:27.014268 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f19ba569-5fb5787844-wnqfr"] Oct 30 01:27:27.018790 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:27:27.018746 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b1bcfa7_d075_4aa9_aca3_afd74f76a125.slice/crio-63690a9d7545ee480e63c11c41f34d7f811d82866c128399b12fb602c7b4a1d2 WatchSource:0}: Error finding container 63690a9d7545ee480e63c11c41f34d7f811d82866c128399b12fb602c7b4a1d2: Status 404 returned error can't find the container with id 63690a9d7545ee480e63c11c41f34d7f811d82866c128399b12fb602c7b4a1d2 Oct 30 01:27:27.785878 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:27.785844 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f19ba569-5fb5787844-wnqfr" event={"ID":"7b1bcfa7-d075-4aa9-aca3-afd74f76a125","Type":"ContainerStarted","Data":"9c3de723b98d993cf182daab6320b1c00f553be3b7bfaff09381f1f848aa8a9c"} Oct 30 01:27:27.786344 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:27.785899 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f19ba569-5fb5787844-wnqfr" event={"ID":"7b1bcfa7-d075-4aa9-aca3-afd74f76a125","Type":"ContainerStarted","Data":"63690a9d7545ee480e63c11c41f34d7f811d82866c128399b12fb602c7b4a1d2"} Oct 30 01:27:27.786344 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:27.786014 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9801220-f94447d96-7sv8l" Oct 30 01:27:27.786344 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:27.786056 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f19ba569-5fb5787844-wnqfr" Oct 30 01:27:27.831724 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:27.831667 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f19ba569-5fb5787844-wnqfr" podStartSLOduration=1.39304644 podStartE2EDuration="1.831648869s" totalImagesPullingTime="438.602429ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:27.829984569 +0000 UTC m=+571.408271906" watchObservedRunningTime="2026-10-30 01:27:27.831648869 +0000 UTC m=+571.409936207" Oct 30 01:27:28.932882 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:28.932839 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:27:30.099104 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:30.099070 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5b724cd8-74fd7b978-7htgh"] Oct 30 01:27:30.102918 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:30.102901 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b724cd8-74fd7b978-7htgh" Oct 30 01:27:30.134017 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:30.133992 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-5b724cd8-serving-cert\"" Oct 30 01:27:30.135549 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:30.135506 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5b724cd8-74fd7b978-7htgh"] Oct 30 01:27:30.142715 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:30.142685 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4e5b900-9733-4b6e-9d10-8284175d21f5-metrics-certs\") pod \"maas-api-e2e-ait-5b724cd8-74fd7b978-7htgh\" (UID: \"e4e5b900-9733-4b6e-9d10-8284175d21f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b724cd8-74fd7b978-7htgh" Oct 30 01:27:30.142870 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:30.142761 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4e5b900-9733-4b6e-9d10-8284175d21f5-openshift-service-ca\") pod \"maas-api-e2e-ait-5b724cd8-74fd7b978-7htgh\" (UID: \"e4e5b900-9733-4b6e-9d10-8284175d21f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b724cd8-74fd7b978-7htgh" Oct 30 01:27:30.142870 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:30.142801 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqfd2\" (UniqueName: \"kubernetes.io/projected/e4e5b900-9733-4b6e-9d10-8284175d21f5-kube-api-access-rqfd2\") pod \"maas-api-e2e-ait-5b724cd8-74fd7b978-7htgh\" (UID: \"e4e5b900-9733-4b6e-9d10-8284175d21f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b724cd8-74fd7b978-7htgh" Oct 30 01:27:30.142870 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:30.142818 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4e5b900-9733-4b6e-9d10-8284175d21f5-maas-api-tls\") pod \"maas-api-e2e-ait-5b724cd8-74fd7b978-7htgh\" (UID: \"e4e5b900-9733-4b6e-9d10-8284175d21f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b724cd8-74fd7b978-7htgh" Oct 30 01:27:30.243658 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:30.243611 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4e5b900-9733-4b6e-9d10-8284175d21f5-metrics-certs\") pod \"maas-api-e2e-ait-5b724cd8-74fd7b978-7htgh\" (UID: \"e4e5b900-9733-4b6e-9d10-8284175d21f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b724cd8-74fd7b978-7htgh" Oct 30 01:27:30.243842 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:30.243690 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4e5b900-9733-4b6e-9d10-8284175d21f5-openshift-service-ca\") pod \"maas-api-e2e-ait-5b724cd8-74fd7b978-7htgh\" (UID: \"e4e5b900-9733-4b6e-9d10-8284175d21f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b724cd8-74fd7b978-7htgh" Oct 30 01:27:30.243842 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:30.243721 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqfd2\" (UniqueName: \"kubernetes.io/projected/e4e5b900-9733-4b6e-9d10-8284175d21f5-kube-api-access-rqfd2\") pod \"maas-api-e2e-ait-5b724cd8-74fd7b978-7htgh\" (UID: \"e4e5b900-9733-4b6e-9d10-8284175d21f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b724cd8-74fd7b978-7htgh" Oct 30 01:27:30.243842 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:30.243743 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4e5b900-9733-4b6e-9d10-8284175d21f5-maas-api-tls\") pod \"maas-api-e2e-ait-5b724cd8-74fd7b978-7htgh\" (UID: \"e4e5b900-9733-4b6e-9d10-8284175d21f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b724cd8-74fd7b978-7htgh" Oct 30 01:27:30.243975 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:27:30.243842 6221 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-5b724cd8-serving-cert: secret "maas-api-e2e-ait-5b724cd8-serving-cert" not found Oct 30 01:27:30.243975 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:27:30.243903 6221 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4e5b900-9733-4b6e-9d10-8284175d21f5-maas-api-tls podName:e4e5b900-9733-4b6e-9d10-8284175d21f5 nodeName:}" failed. No retries permitted until 2026-10-30 01:27:30.743886517 +0000 UTC m=+574.322173844 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e4e5b900-9733-4b6e-9d10-8284175d21f5-maas-api-tls") pod "maas-api-e2e-ait-5b724cd8-74fd7b978-7htgh" (UID: "e4e5b900-9733-4b6e-9d10-8284175d21f5") : secret "maas-api-e2e-ait-5b724cd8-serving-cert" not found Oct 30 01:27:30.244458 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:30.244433 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4e5b900-9733-4b6e-9d10-8284175d21f5-openshift-service-ca\") pod \"maas-api-e2e-ait-5b724cd8-74fd7b978-7htgh\" (UID: \"e4e5b900-9733-4b6e-9d10-8284175d21f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b724cd8-74fd7b978-7htgh" Oct 30 01:27:30.246059 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:30.246037 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4e5b900-9733-4b6e-9d10-8284175d21f5-metrics-certs\") pod \"maas-api-e2e-ait-5b724cd8-74fd7b978-7htgh\" (UID: \"e4e5b900-9733-4b6e-9d10-8284175d21f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b724cd8-74fd7b978-7htgh" Oct 30 01:27:30.272652 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:30.272625 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqfd2\" (UniqueName: \"kubernetes.io/projected/e4e5b900-9733-4b6e-9d10-8284175d21f5-kube-api-access-rqfd2\") pod \"maas-api-e2e-ait-5b724cd8-74fd7b978-7htgh\" (UID: \"e4e5b900-9733-4b6e-9d10-8284175d21f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b724cd8-74fd7b978-7htgh" Oct 30 01:27:30.752472 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:30.752429 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4e5b900-9733-4b6e-9d10-8284175d21f5-maas-api-tls\") pod \"maas-api-e2e-ait-5b724cd8-74fd7b978-7htgh\" (UID: \"e4e5b900-9733-4b6e-9d10-8284175d21f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b724cd8-74fd7b978-7htgh" Oct 30 01:27:30.754775 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:30.754751 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4e5b900-9733-4b6e-9d10-8284175d21f5-maas-api-tls\") pod \"maas-api-e2e-ait-5b724cd8-74fd7b978-7htgh\" (UID: \"e4e5b900-9733-4b6e-9d10-8284175d21f5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b724cd8-74fd7b978-7htgh" Oct 30 01:27:31.016101 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:31.016022 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b724cd8-74fd7b978-7htgh" Oct 30 01:27:31.345249 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:31.345214 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5b724cd8-74fd7b978-7htgh"] Oct 30 01:27:31.348740 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:27:31.348706 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4e5b900_9733_4b6e_9d10_8284175d21f5.slice/crio-620a11864e74a82d39c6601808b0a6784747318fadf87d72f8053520626cbeb7 WatchSource:0}: Error finding container 620a11864e74a82d39c6601808b0a6784747318fadf87d72f8053520626cbeb7: Status 404 returned error can't find the container with id 620a11864e74a82d39c6601808b0a6784747318fadf87d72f8053520626cbeb7 Oct 30 01:27:31.845213 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:31.845135 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b724cd8-74fd7b978-7htgh" event={"ID":"e4e5b900-9733-4b6e-9d10-8284175d21f5","Type":"ContainerStarted","Data":"620a11864e74a82d39c6601808b0a6784747318fadf87d72f8053520626cbeb7"} Oct 30 01:27:32.141124 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:32.141037 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f29772b9-c9bbdbc8f-tkpwz"] Oct 30 01:27:32.144659 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:32.144643 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f29772b9-c9bbdbc8f-tkpwz" Oct 30 01:27:32.174460 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:32.174275 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f29772b9-serving-cert\"" Oct 30 01:27:32.174757 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:32.174501 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdmfh\" (UniqueName: \"kubernetes.io/projected/58489c7b-813d-488b-936f-0772735cb950-kube-api-access-mdmfh\") pod \"maas-api-e2e-ait-f29772b9-c9bbdbc8f-tkpwz\" (UID: \"58489c7b-813d-488b-936f-0772735cb950\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f29772b9-c9bbdbc8f-tkpwz" Oct 30 01:27:32.174757 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:32.174633 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58489c7b-813d-488b-936f-0772735cb950-maas-api-tls\") pod \"maas-api-e2e-ait-f29772b9-c9bbdbc8f-tkpwz\" (UID: \"58489c7b-813d-488b-936f-0772735cb950\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f29772b9-c9bbdbc8f-tkpwz" Oct 30 01:27:32.174907 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:32.174788 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58489c7b-813d-488b-936f-0772735cb950-openshift-service-ca\") pod \"maas-api-e2e-ait-f29772b9-c9bbdbc8f-tkpwz\" (UID: \"58489c7b-813d-488b-936f-0772735cb950\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f29772b9-c9bbdbc8f-tkpwz" Oct 30 01:27:32.174907 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:32.174818 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58489c7b-813d-488b-936f-0772735cb950-metrics-certs\") pod \"maas-api-e2e-ait-f29772b9-c9bbdbc8f-tkpwz\" (UID: \"58489c7b-813d-488b-936f-0772735cb950\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f29772b9-c9bbdbc8f-tkpwz" Oct 30 01:27:32.176886 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:32.176863 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f29772b9-c9bbdbc8f-tkpwz"] Oct 30 01:27:32.275937 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:32.275898 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58489c7b-813d-488b-936f-0772735cb950-maas-api-tls\") pod \"maas-api-e2e-ait-f29772b9-c9bbdbc8f-tkpwz\" (UID: \"58489c7b-813d-488b-936f-0772735cb950\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f29772b9-c9bbdbc8f-tkpwz" Oct 30 01:27:32.276125 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:32.275961 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58489c7b-813d-488b-936f-0772735cb950-openshift-service-ca\") pod \"maas-api-e2e-ait-f29772b9-c9bbdbc8f-tkpwz\" (UID: \"58489c7b-813d-488b-936f-0772735cb950\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f29772b9-c9bbdbc8f-tkpwz" Oct 30 01:27:32.276125 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:32.275989 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58489c7b-813d-488b-936f-0772735cb950-metrics-certs\") pod \"maas-api-e2e-ait-f29772b9-c9bbdbc8f-tkpwz\" (UID: \"58489c7b-813d-488b-936f-0772735cb950\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f29772b9-c9bbdbc8f-tkpwz" Oct 30 01:27:32.276125 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:32.276027 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdmfh\" (UniqueName: \"kubernetes.io/projected/58489c7b-813d-488b-936f-0772735cb950-kube-api-access-mdmfh\") pod \"maas-api-e2e-ait-f29772b9-c9bbdbc8f-tkpwz\" (UID: \"58489c7b-813d-488b-936f-0772735cb950\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f29772b9-c9bbdbc8f-tkpwz" Oct 30 01:27:32.276751 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:32.276726 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58489c7b-813d-488b-936f-0772735cb950-openshift-service-ca\") pod \"maas-api-e2e-ait-f29772b9-c9bbdbc8f-tkpwz\" (UID: \"58489c7b-813d-488b-936f-0772735cb950\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f29772b9-c9bbdbc8f-tkpwz" Oct 30 01:27:32.278462 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:32.278441 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58489c7b-813d-488b-936f-0772735cb950-maas-api-tls\") pod \"maas-api-e2e-ait-f29772b9-c9bbdbc8f-tkpwz\" (UID: \"58489c7b-813d-488b-936f-0772735cb950\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f29772b9-c9bbdbc8f-tkpwz" Oct 30 01:27:32.278569 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:32.278496 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58489c7b-813d-488b-936f-0772735cb950-metrics-certs\") pod \"maas-api-e2e-ait-f29772b9-c9bbdbc8f-tkpwz\" (UID: \"58489c7b-813d-488b-936f-0772735cb950\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f29772b9-c9bbdbc8f-tkpwz" Oct 30 01:27:32.294900 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:32.294881 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdmfh\" (UniqueName: \"kubernetes.io/projected/58489c7b-813d-488b-936f-0772735cb950-kube-api-access-mdmfh\") pod \"maas-api-e2e-ait-f29772b9-c9bbdbc8f-tkpwz\" (UID: \"58489c7b-813d-488b-936f-0772735cb950\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f29772b9-c9bbdbc8f-tkpwz" Oct 30 01:27:32.457989 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:32.457911 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f29772b9-c9bbdbc8f-tkpwz" Oct 30 01:27:32.754542 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:32.754456 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f29772b9-c9bbdbc8f-tkpwz"] Oct 30 01:27:32.758161 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:27:32.758130 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58489c7b_813d_488b_936f_0772735cb950.slice/crio-5a5991559ece16abfa2b59e74348d170911b4063013ee031f2f8bb2e2e03ec77 WatchSource:0}: Error finding container 5a5991559ece16abfa2b59e74348d170911b4063013ee031f2f8bb2e2e03ec77: Status 404 returned error can't find the container with id 5a5991559ece16abfa2b59e74348d170911b4063013ee031f2f8bb2e2e03ec77 Oct 30 01:27:32.850554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:32.850504 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f29772b9-c9bbdbc8f-tkpwz" event={"ID":"58489c7b-813d-488b-936f-0772735cb950","Type":"ContainerStarted","Data":"5a5991559ece16abfa2b59e74348d170911b4063013ee031f2f8bb2e2e03ec77"} Oct 30 01:27:32.852158 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:32.852134 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b724cd8-74fd7b978-7htgh" event={"ID":"e4e5b900-9733-4b6e-9d10-8284175d21f5","Type":"ContainerStarted","Data":"f42c0b7a616e50554f67c98b89111125f2d210bd1a1299a8bd81b2f3d360299a"} Oct 30 01:27:32.852280 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:32.852257 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b724cd8-74fd7b978-7htgh" Oct 30 01:27:32.891844 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:32.891796 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b724cd8-74fd7b978-7htgh" podStartSLOduration=2.4462727 podStartE2EDuration="2.891783927s" totalImagesPullingTime="445.511227ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:32.891436534 +0000 UTC m=+576.469723869" watchObservedRunningTime="2026-10-30 01:27:32.891783927 +0000 UTC m=+576.470071263" Oct 30 01:27:33.795716 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:33.795688 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f19ba569-5fb5787844-wnqfr" Oct 30 01:27:33.796105 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:33.795754 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9801220-f94447d96-7sv8l" Oct 30 01:27:33.859975 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:33.859942 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f29772b9-c9bbdbc8f-tkpwz" event={"ID":"58489c7b-813d-488b-936f-0772735cb950","Type":"ContainerStarted","Data":"55b292f557a6794f88ed73cf0e7aed5a8231e61a9b9d213675711a41ae88cfc9"} Oct 30 01:27:33.860163 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:33.860091 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f29772b9-c9bbdbc8f-tkpwz" Oct 30 01:27:33.918165 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:33.918104 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f29772b9-c9bbdbc8f-tkpwz" podStartSLOduration=1.54824999 podStartE2EDuration="1.918083368s" totalImagesPullingTime="369.833378ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:33.915922851 +0000 UTC m=+577.494210199" watchObservedRunningTime="2026-10-30 01:27:33.918083368 +0000 UTC m=+577.496370705" Oct 30 01:27:34.165326 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:34.165227 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:27:35.773493 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:35.773459 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d9801220-6c74684cd7-4x5z4" Oct 30 01:27:35.773949 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:35.773542 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-d9801220-7df68db76d-zrpzc" Oct 30 01:27:36.781756 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:36.781725 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-5b724cd8-6fd679fd8-phz5k" Oct 30 01:27:38.866023 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:38.865989 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b724cd8-74fd7b978-7htgh" Oct 30 01:27:38.894670 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:38.894638 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-18c05df8-7b97d9bf6f-q95mm"] Oct 30 01:27:38.899075 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:38.899060 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-18c05df8-7b97d9bf6f-q95mm" Oct 30 01:27:38.926874 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:38.926841 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-18c05df8-serving-cert\"" Oct 30 01:27:38.942067 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:38.942034 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-18c05df8-7b97d9bf6f-q95mm"] Oct 30 01:27:38.958929 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:38.958890 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tfrn\" (UniqueName: \"kubernetes.io/projected/cb7fd5cb-4d81-450e-bb74-ae198f698559-kube-api-access-6tfrn\") pod \"maas-api-e2e-derive-18c05df8-7b97d9bf6f-q95mm\" (UID: \"cb7fd5cb-4d81-450e-bb74-ae198f698559\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-18c05df8-7b97d9bf6f-q95mm" Oct 30 01:27:38.959293 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:38.959271 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb7fd5cb-4d81-450e-bb74-ae198f698559-metrics-certs\") pod \"maas-api-e2e-derive-18c05df8-7b97d9bf6f-q95mm\" (UID: \"cb7fd5cb-4d81-450e-bb74-ae198f698559\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-18c05df8-7b97d9bf6f-q95mm" Oct 30 01:27:38.959559 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:38.959513 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb7fd5cb-4d81-450e-bb74-ae198f698559-maas-api-tls\") pod \"maas-api-e2e-derive-18c05df8-7b97d9bf6f-q95mm\" (UID: \"cb7fd5cb-4d81-450e-bb74-ae198f698559\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-18c05df8-7b97d9bf6f-q95mm" Oct 30 01:27:38.960001 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:38.959979 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb7fd5cb-4d81-450e-bb74-ae198f698559-openshift-service-ca\") pod \"maas-api-e2e-derive-18c05df8-7b97d9bf6f-q95mm\" (UID: \"cb7fd5cb-4d81-450e-bb74-ae198f698559\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-18c05df8-7b97d9bf6f-q95mm" Oct 30 01:27:39.061636 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:39.061332 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb7fd5cb-4d81-450e-bb74-ae198f698559-openshift-service-ca\") pod \"maas-api-e2e-derive-18c05df8-7b97d9bf6f-q95mm\" (UID: \"cb7fd5cb-4d81-450e-bb74-ae198f698559\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-18c05df8-7b97d9bf6f-q95mm" Oct 30 01:27:39.061636 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:39.061406 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tfrn\" (UniqueName: \"kubernetes.io/projected/cb7fd5cb-4d81-450e-bb74-ae198f698559-kube-api-access-6tfrn\") pod \"maas-api-e2e-derive-18c05df8-7b97d9bf6f-q95mm\" (UID: \"cb7fd5cb-4d81-450e-bb74-ae198f698559\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-18c05df8-7b97d9bf6f-q95mm" Oct 30 01:27:39.061636 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:39.061475 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb7fd5cb-4d81-450e-bb74-ae198f698559-metrics-certs\") pod \"maas-api-e2e-derive-18c05df8-7b97d9bf6f-q95mm\" (UID: \"cb7fd5cb-4d81-450e-bb74-ae198f698559\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-18c05df8-7b97d9bf6f-q95mm" Oct 30 01:27:39.061636 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:39.061559 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb7fd5cb-4d81-450e-bb74-ae198f698559-maas-api-tls\") pod \"maas-api-e2e-derive-18c05df8-7b97d9bf6f-q95mm\" (UID: \"cb7fd5cb-4d81-450e-bb74-ae198f698559\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-18c05df8-7b97d9bf6f-q95mm" Oct 30 01:27:39.061975 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:27:39.061701 6221 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-18c05df8-serving-cert: secret "maas-api-e2e-derive-18c05df8-serving-cert" not found Oct 30 01:27:39.061975 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:27:39.061784 6221 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb7fd5cb-4d81-450e-bb74-ae198f698559-maas-api-tls podName:cb7fd5cb-4d81-450e-bb74-ae198f698559 nodeName:}" failed. No retries permitted until 2026-10-30 01:27:39.561763865 +0000 UTC m=+583.140051202 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cb7fd5cb-4d81-450e-bb74-ae198f698559-maas-api-tls") pod "maas-api-e2e-derive-18c05df8-7b97d9bf6f-q95mm" (UID: "cb7fd5cb-4d81-450e-bb74-ae198f698559") : secret "maas-api-e2e-derive-18c05df8-serving-cert" not found Oct 30 01:27:39.062188 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:39.062165 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb7fd5cb-4d81-450e-bb74-ae198f698559-openshift-service-ca\") pod \"maas-api-e2e-derive-18c05df8-7b97d9bf6f-q95mm\" (UID: \"cb7fd5cb-4d81-450e-bb74-ae198f698559\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-18c05df8-7b97d9bf6f-q95mm" Oct 30 01:27:39.064232 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:39.064208 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb7fd5cb-4d81-450e-bb74-ae198f698559-metrics-certs\") pod \"maas-api-e2e-derive-18c05df8-7b97d9bf6f-q95mm\" (UID: \"cb7fd5cb-4d81-450e-bb74-ae198f698559\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-18c05df8-7b97d9bf6f-q95mm" Oct 30 01:27:39.134449 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:39.134378 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tfrn\" (UniqueName: \"kubernetes.io/projected/cb7fd5cb-4d81-450e-bb74-ae198f698559-kube-api-access-6tfrn\") pod \"maas-api-e2e-derive-18c05df8-7b97d9bf6f-q95mm\" (UID: \"cb7fd5cb-4d81-450e-bb74-ae198f698559\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-18c05df8-7b97d9bf6f-q95mm" Oct 30 01:27:39.567449 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:39.567416 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb7fd5cb-4d81-450e-bb74-ae198f698559-maas-api-tls\") pod \"maas-api-e2e-derive-18c05df8-7b97d9bf6f-q95mm\" (UID: \"cb7fd5cb-4d81-450e-bb74-ae198f698559\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-18c05df8-7b97d9bf6f-q95mm" Oct 30 01:27:39.569759 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:39.569733 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb7fd5cb-4d81-450e-bb74-ae198f698559-maas-api-tls\") pod \"maas-api-e2e-derive-18c05df8-7b97d9bf6f-q95mm\" (UID: \"cb7fd5cb-4d81-450e-bb74-ae198f698559\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-18c05df8-7b97d9bf6f-q95mm" Oct 30 01:27:39.846082 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:39.845985 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-18c05df8-7b97d9bf6f-q95mm" Oct 30 01:27:39.878830 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:39.878801 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f29772b9-c9bbdbc8f-tkpwz" Oct 30 01:27:40.242603 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:40.225829 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-18c05df8-7b97d9bf6f-q95mm"] Oct 30 01:27:40.260772 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:27:40.260718 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb7fd5cb_4d81_450e_bb74_ae198f698559.slice/crio-ae00f5b0238c5907525bc7a9d5958526c0bb8c8facbbf38bfd803b86519e966f WatchSource:0}: Error finding container ae00f5b0238c5907525bc7a9d5958526c0bb8c8facbbf38bfd803b86519e966f: Status 404 returned error can't find the container with id ae00f5b0238c5907525bc7a9d5958526c0bb8c8facbbf38bfd803b86519e966f Oct 30 01:27:40.895561 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:40.895512 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-18c05df8-7b97d9bf6f-q95mm" event={"ID":"cb7fd5cb-4d81-450e-bb74-ae198f698559","Type":"ContainerStarted","Data":"ae00f5b0238c5907525bc7a9d5958526c0bb8c8facbbf38bfd803b86519e966f"} Oct 30 01:27:41.902146 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:41.902101 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-18c05df8-7b97d9bf6f-q95mm" event={"ID":"cb7fd5cb-4d81-450e-bb74-ae198f698559","Type":"ContainerStarted","Data":"a958a37f507527a9b113110e465377c36bed619439ef7ae89aeb77c88e5af261"} Oct 30 01:27:41.902633 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:41.902261 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-18c05df8-7b97d9bf6f-q95mm" Oct 30 01:27:41.943811 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:41.943757 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-18c05df8-7b97d9bf6f-q95mm" podStartSLOduration=3.415406797 podStartE2EDuration="3.943742365s" totalImagesPullingTime="528.335568ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:41.941917238 +0000 UTC m=+585.520204556" watchObservedRunningTime="2026-10-30 01:27:41.943742365 +0000 UTC m=+585.522029721" Oct 30 01:27:42.354743 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:42.354709 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:27:44.939200 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:44.939111 6221 generic.go:356] "Generic (PLEG): container finished" podID="ec361aac-99e4-4f3c-9c0a-33411f8f74e3" containerID="24ede9319bbff330d24a8f9e1c538c3dbfd69f8bf0c6c448f05c44837902ef50" exitCode=0 Oct 30 01:27:44.939200 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:44.939174 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" event={"ID":"ec361aac-99e4-4f3c-9c0a-33411f8f74e3","Type":"ContainerDied","Data":"24ede9319bbff330d24a8f9e1c538c3dbfd69f8bf0c6c448f05c44837902ef50"} Oct 30 01:27:44.939816 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:44.939578 6221 scope.go:122] "RemoveContainer" containerID="24ede9319bbff330d24a8f9e1c538c3dbfd69f8bf0c6c448f05c44837902ef50" Oct 30 01:27:44.940846 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:44.940818 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:27:44.940940 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:44.940886 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:27:44.940940 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:44.940915 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:27:45.947386 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:45.947345 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" event={"ID":"ec361aac-99e4-4f3c-9c0a-33411f8f74e3","Type":"ContainerStarted","Data":"e2e40b5f725217a45cd267db9bba839c6b2665ca2fa0b45028aabd990910324b"} Oct 30 01:27:46.564400 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:46.564368 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" Oct 30 01:27:46.565664 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:46.565636 6221 patch_prober.go:29] interesting pod/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 01:27:46.565788 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:46.565684 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" podUID="ec361aac-99e4-4f3c-9c0a-33411f8f74e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 01:27:47.196026 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:47.195981 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d9801220-7df68db76d-zrpzc"] Oct 30 01:27:47.234234 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:47.234194 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d9801220-6c74684cd7-4x5z4"] Oct 30 01:27:47.565001 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:47.564956 6221 patch_prober.go:29] interesting pod/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 01:27:47.565180 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:47.565028 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" podUID="ec361aac-99e4-4f3c-9c0a-33411f8f74e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 01:27:47.934876 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:47.934412 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-18c05df8-7b97d9bf6f-q95mm" Oct 30 01:27:48.564731 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:48.564700 6221 patch_prober.go:29] interesting pod/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 01:27:48.565138 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:48.564755 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" podUID="ec361aac-99e4-4f3c-9c0a-33411f8f74e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 01:27:48.962713 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:48.962678 6221 generic.go:356] "Generic (PLEG): container finished" podID="03ccf6ef-c34a-4847-9791-1050db4d14d5" containerID="f2ac4d800448acebbc51e1f8e6ab4063ade4400929aeecef50b9a112deaa126c" exitCode=0 Oct 30 01:27:48.962946 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:48.962754 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" event={"ID":"03ccf6ef-c34a-4847-9791-1050db4d14d5","Type":"ContainerDied","Data":"f2ac4d800448acebbc51e1f8e6ab4063ade4400929aeecef50b9a112deaa126c"} Oct 30 01:27:48.963199 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:48.963183 6221 scope.go:122] "RemoveContainer" containerID="f2ac4d800448acebbc51e1f8e6ab4063ade4400929aeecef50b9a112deaa126c" Oct 30 01:27:48.964437 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:48.964411 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:27:48.964490 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:48.964477 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:27:48.964524 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:48.964509 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:27:49.565672 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:49.565634 6221 patch_prober.go:29] interesting pod/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 01:27:49.566086 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:49.565687 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" podUID="ec361aac-99e4-4f3c-9c0a-33411f8f74e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 01:27:49.969179 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:49.969092 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" event={"ID":"03ccf6ef-c34a-4847-9791-1050db4d14d5","Type":"ContainerStarted","Data":"b92a681a2f3ca74e300f3cd73efe4201ad8a396a01097a2fdad2af7f7e86c1ec"} Oct 30 01:27:50.565012 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:50.564968 6221 patch_prober.go:29] interesting pod/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Oct 30 01:27:50.565192 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:50.565026 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" podUID="ec361aac-99e4-4f3c-9c0a-33411f8f74e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Oct 30 01:27:50.743553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:50.742920 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" Oct 30 01:27:50.773022 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:50.772990 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" Oct 30 01:27:50.774248 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:50.774226 6221 patch_prober.go:29] interesting pod/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 01:27:50.774352 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:50.774269 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" podUID="03ccf6ef-c34a-4847-9791-1050db4d14d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 01:27:51.493743 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:27:51.493701 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:27:51.569092 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:51.569062 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" Oct 30 01:27:51.569965 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:51.569947 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" Oct 30 01:27:51.746218 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:51.746135 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" Oct 30 01:27:51.773423 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:51.773393 6221 patch_prober.go:29] interesting pod/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 01:27:51.773582 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:51.773450 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" podUID="03ccf6ef-c34a-4847-9791-1050db4d14d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 01:27:52.197208 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:52.197140 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-d9801220-7df68db76d-zrpzc" podUID="2e4c7a82-6d00-4b93-bad8-f20425f04d40" containerName="payload-processing" containerID="cri-o://201637b89e0f197a1b6748842ac49b767c952d3282255117385f56497b9e4ad9" gracePeriod=25 Oct 30 01:27:52.234895 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:52.234848 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d9801220-6c74684cd7-4x5z4" podUID="f0d2d13d-4eb6-4a8a-8604-d893dbfe03df" containerName="payload-pre-processing" containerID="cri-o://6485652b905165fc5ef851f8543d2cecc7dadb640a8b324be248753bab75de59" gracePeriod=25 Oct 30 01:27:52.539468 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:52.539224 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d9801220-7df68db76d-zrpzc" Oct 30 01:27:52.586180 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:52.586152 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d9801220-6c74684cd7-4x5z4" Oct 30 01:27:52.685826 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:52.685792 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e4c7a82-6d00-4b93-bad8-f20425f04d40-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e4c7a82-6d00-4b93-bad8-f20425f04d40-plugins-config-volume\") pod \"2e4c7a82-6d00-4b93-bad8-f20425f04d40\" (UID: \"2e4c7a82-6d00-4b93-bad8-f20425f04d40\") " Oct 30 01:27:52.686011 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:52.685844 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e4c7a82-6d00-4b93-bad8-f20425f04d40-kube-api-access-pfk7j\" (UniqueName: \"kubernetes.io/projected/2e4c7a82-6d00-4b93-bad8-f20425f04d40-kube-api-access-pfk7j\") pod \"2e4c7a82-6d00-4b93-bad8-f20425f04d40\" (UID: \"2e4c7a82-6d00-4b93-bad8-f20425f04d40\") " Oct 30 01:27:52.686011 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:52.685977 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0d2d13d-4eb6-4a8a-8604-d893dbfe03df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0d2d13d-4eb6-4a8a-8604-d893dbfe03df-plugins-config-volume\") pod \"f0d2d13d-4eb6-4a8a-8604-d893dbfe03df\" (UID: \"f0d2d13d-4eb6-4a8a-8604-d893dbfe03df\") " Oct 30 01:27:52.686125 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:52.686032 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0d2d13d-4eb6-4a8a-8604-d893dbfe03df-kube-api-access-s2xzf\" (UniqueName: \"kubernetes.io/projected/f0d2d13d-4eb6-4a8a-8604-d893dbfe03df-kube-api-access-s2xzf\") pod \"f0d2d13d-4eb6-4a8a-8604-d893dbfe03df\" (UID: \"f0d2d13d-4eb6-4a8a-8604-d893dbfe03df\") " Oct 30 01:27:52.686180 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:52.686153 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e4c7a82-6d00-4b93-bad8-f20425f04d40-plugins-config-volume" pod "2e4c7a82-6d00-4b93-bad8-f20425f04d40" (UID: "2e4c7a82-6d00-4b93-bad8-f20425f04d40"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:27:52.686376 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:52.686356 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0d2d13d-4eb6-4a8a-8604-d893dbfe03df-plugins-config-volume" pod "f0d2d13d-4eb6-4a8a-8604-d893dbfe03df" (UID: "f0d2d13d-4eb6-4a8a-8604-d893dbfe03df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:27:52.686457 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:52.686443 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e4c7a82-6d00-4b93-bad8-f20425f04d40-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:27:52.686491 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:52.686459 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0d2d13d-4eb6-4a8a-8604-d893dbfe03df-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:27:52.687979 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:52.687953 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e4c7a82-6d00-4b93-bad8-f20425f04d40-kube-api-access-pfk7j" pod "2e4c7a82-6d00-4b93-bad8-f20425f04d40" (UID: "2e4c7a82-6d00-4b93-bad8-f20425f04d40"). InnerVolumeSpecName "kube-api-access-pfk7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:52.687979 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:52.687965 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0d2d13d-4eb6-4a8a-8604-d893dbfe03df-kube-api-access-s2xzf" pod "f0d2d13d-4eb6-4a8a-8604-d893dbfe03df" (UID: "f0d2d13d-4eb6-4a8a-8604-d893dbfe03df"). InnerVolumeSpecName "kube-api-access-s2xzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:52.773887 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:52.773810 6221 patch_prober.go:29] interesting pod/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 01:27:52.773887 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:52.773869 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" podUID="03ccf6ef-c34a-4847-9791-1050db4d14d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 01:27:52.787371 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:52.787346 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfk7j\" (UniqueName: \"kubernetes.io/projected/2e4c7a82-6d00-4b93-bad8-f20425f04d40-kube-api-access-pfk7j\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:27:52.787371 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:52.787367 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2xzf\" (UniqueName: \"kubernetes.io/projected/f0d2d13d-4eb6-4a8a-8604-d893dbfe03df-kube-api-access-s2xzf\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:27:52.987491 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:52.987456 6221 generic.go:356] "Generic (PLEG): container finished" podID="2e4c7a82-6d00-4b93-bad8-f20425f04d40" containerID="201637b89e0f197a1b6748842ac49b767c952d3282255117385f56497b9e4ad9" exitCode=0 Oct 30 01:27:52.987704 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:52.987505 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d9801220-7df68db76d-zrpzc" Oct 30 01:27:52.987704 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:52.987567 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d9801220-7df68db76d-zrpzc" event={"ID":"2e4c7a82-6d00-4b93-bad8-f20425f04d40","Type":"ContainerDied","Data":"201637b89e0f197a1b6748842ac49b767c952d3282255117385f56497b9e4ad9"} Oct 30 01:27:52.987704 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:52.987628 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d9801220-7df68db76d-zrpzc" event={"ID":"2e4c7a82-6d00-4b93-bad8-f20425f04d40","Type":"ContainerDied","Data":"ad57a65e02c18f73a4a2ad83374a0f3a4bd512283e43d9611f2d0eb549e53433"} Oct 30 01:27:52.987704 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:52.987659 6221 scope.go:122] "RemoveContainer" containerID="201637b89e0f197a1b6748842ac49b767c952d3282255117385f56497b9e4ad9" Oct 30 01:27:52.988950 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:52.988918 6221 generic.go:356] "Generic (PLEG): container finished" podID="f0d2d13d-4eb6-4a8a-8604-d893dbfe03df" containerID="6485652b905165fc5ef851f8543d2cecc7dadb640a8b324be248753bab75de59" exitCode=0 Oct 30 01:27:52.989048 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:52.988974 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d9801220-6c74684cd7-4x5z4" Oct 30 01:27:52.989048 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:52.988971 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d9801220-6c74684cd7-4x5z4" event={"ID":"f0d2d13d-4eb6-4a8a-8604-d893dbfe03df","Type":"ContainerDied","Data":"6485652b905165fc5ef851f8543d2cecc7dadb640a8b324be248753bab75de59"} Oct 30 01:27:52.989166 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:52.989066 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d9801220-6c74684cd7-4x5z4" event={"ID":"f0d2d13d-4eb6-4a8a-8604-d893dbfe03df","Type":"ContainerDied","Data":"572ea634f837bee0052600cb5d85abcbf9dc1d7534bded1fd2e3f52cbaa5eb96"} Oct 30 01:27:52.997807 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:52.997660 6221 scope.go:122] "RemoveContainer" containerID="201637b89e0f197a1b6748842ac49b767c952d3282255117385f56497b9e4ad9" Oct 30 01:27:52.997956 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:27:52.997936 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"201637b89e0f197a1b6748842ac49b767c952d3282255117385f56497b9e4ad9\": container with ID starting with 201637b89e0f197a1b6748842ac49b767c952d3282255117385f56497b9e4ad9 not found: ID does not exist" containerID="201637b89e0f197a1b6748842ac49b767c952d3282255117385f56497b9e4ad9" Oct 30 01:27:52.998030 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:52.997963 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"201637b89e0f197a1b6748842ac49b767c952d3282255117385f56497b9e4ad9"} err="failed to get container status \"201637b89e0f197a1b6748842ac49b767c952d3282255117385f56497b9e4ad9\": rpc error: code = NotFound desc = could not find container \"201637b89e0f197a1b6748842ac49b767c952d3282255117385f56497b9e4ad9\": container with ID starting with 201637b89e0f197a1b6748842ac49b767c952d3282255117385f56497b9e4ad9 not found: ID does not exist" Oct 30 01:27:52.998030 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:52.997978 6221 scope.go:122] "RemoveContainer" containerID="6485652b905165fc5ef851f8543d2cecc7dadb640a8b324be248753bab75de59" Oct 30 01:27:53.032867 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:53.032735 6221 scope.go:122] "RemoveContainer" containerID="6485652b905165fc5ef851f8543d2cecc7dadb640a8b324be248753bab75de59" Oct 30 01:27:53.033391 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:27:53.033348 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6485652b905165fc5ef851f8543d2cecc7dadb640a8b324be248753bab75de59\": container with ID starting with 6485652b905165fc5ef851f8543d2cecc7dadb640a8b324be248753bab75de59 not found: ID does not exist" containerID="6485652b905165fc5ef851f8543d2cecc7dadb640a8b324be248753bab75de59" Oct 30 01:27:53.033520 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:53.033403 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6485652b905165fc5ef851f8543d2cecc7dadb640a8b324be248753bab75de59"} err="failed to get container status \"6485652b905165fc5ef851f8543d2cecc7dadb640a8b324be248753bab75de59\": rpc error: code = NotFound desc = could not find container \"6485652b905165fc5ef851f8543d2cecc7dadb640a8b324be248753bab75de59\": container with ID starting with 6485652b905165fc5ef851f8543d2cecc7dadb640a8b324be248753bab75de59 not found: ID does not exist" Oct 30 01:27:53.055731 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:53.055701 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d9801220-7df68db76d-zrpzc"] Oct 30 01:27:53.062502 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:53.062478 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d9801220-7df68db76d-zrpzc"] Oct 30 01:27:53.095919 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:53.095891 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d9801220-6c74684cd7-4x5z4"] Oct 30 01:27:53.101855 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:53.101833 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d9801220-6c74684cd7-4x5z4"] Oct 30 01:27:53.773366 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:53.773332 6221 patch_prober.go:29] interesting pod/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 01:27:53.773566 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:53.773393 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" podUID="03ccf6ef-c34a-4847-9791-1050db4d14d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 01:27:54.231057 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:54.231000 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d9801220-f94447d96-7sv8l"] Oct 30 01:27:54.231906 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:54.231870 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9801220-f94447d96-7sv8l" podUID="b219867f-db12-4b12-918a-24d4151ea2b7" containerName="maas-api" containerID="cri-o://fbaca07c09731ac3325b0496786c2cc36c1120d56290a61b61f8788c0a7b39f7" gracePeriod=30 Oct 30 01:27:54.588809 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:54.588760 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e4c7a82-6d00-4b93-bad8-f20425f04d40" path="/var/lib/kubelet/pods/2e4c7a82-6d00-4b93-bad8-f20425f04d40/volumes" Oct 30 01:27:54.589435 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:54.589345 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0d2d13d-4eb6-4a8a-8604-d893dbfe03df" path="/var/lib/kubelet/pods/f0d2d13d-4eb6-4a8a-8604-d893dbfe03df/volumes" Oct 30 01:27:54.592774 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:54.592751 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9801220-f94447d96-7sv8l" Oct 30 01:27:54.639278 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:54.638952 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f19ba569-5fb5787844-wnqfr"] Oct 30 01:27:54.639278 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:54.639200 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f19ba569-5fb5787844-wnqfr" podUID="7b1bcfa7-d075-4aa9-aca3-afd74f76a125" containerName="maas-api" containerID="cri-o://9c3de723b98d993cf182daab6320b1c00f553be3b7bfaff09381f1f848aa8a9c" gracePeriod=30 Oct 30 01:27:54.708170 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:54.706990 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b219867f-db12-4b12-918a-24d4151ea2b7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b219867f-db12-4b12-918a-24d4151ea2b7-metrics-certs\") pod \"b219867f-db12-4b12-918a-24d4151ea2b7\" (UID: \"b219867f-db12-4b12-918a-24d4151ea2b7\") " Oct 30 01:27:54.708170 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:54.707058 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b219867f-db12-4b12-918a-24d4151ea2b7-kube-api-access-67cll\" (UniqueName: \"kubernetes.io/projected/b219867f-db12-4b12-918a-24d4151ea2b7-kube-api-access-67cll\") pod \"b219867f-db12-4b12-918a-24d4151ea2b7\" (UID: \"b219867f-db12-4b12-918a-24d4151ea2b7\") " Oct 30 01:27:54.708170 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:54.707172 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b219867f-db12-4b12-918a-24d4151ea2b7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b219867f-db12-4b12-918a-24d4151ea2b7-openshift-service-ca\") pod \"b219867f-db12-4b12-918a-24d4151ea2b7\" (UID: \"b219867f-db12-4b12-918a-24d4151ea2b7\") " Oct 30 01:27:54.708170 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:54.707222 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b219867f-db12-4b12-918a-24d4151ea2b7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b219867f-db12-4b12-918a-24d4151ea2b7-maas-api-tls\") pod \"b219867f-db12-4b12-918a-24d4151ea2b7\" (UID: \"b219867f-db12-4b12-918a-24d4151ea2b7\") " Oct 30 01:27:54.711117 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:54.710837 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b219867f-db12-4b12-918a-24d4151ea2b7-metrics-certs" pod "b219867f-db12-4b12-918a-24d4151ea2b7" (UID: "b219867f-db12-4b12-918a-24d4151ea2b7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:27:54.713423 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:54.713222 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b219867f-db12-4b12-918a-24d4151ea2b7-kube-api-access-67cll" pod "b219867f-db12-4b12-918a-24d4151ea2b7" (UID: "b219867f-db12-4b12-918a-24d4151ea2b7"). InnerVolumeSpecName "kube-api-access-67cll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:54.713937 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:54.713692 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b219867f-db12-4b12-918a-24d4151ea2b7-openshift-service-ca" pod "b219867f-db12-4b12-918a-24d4151ea2b7" (UID: "b219867f-db12-4b12-918a-24d4151ea2b7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:27:54.716095 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:54.716063 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b219867f-db12-4b12-918a-24d4151ea2b7-maas-api-tls" pod "b219867f-db12-4b12-918a-24d4151ea2b7" (UID: "b219867f-db12-4b12-918a-24d4151ea2b7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:27:54.774153 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:54.774114 6221 patch_prober.go:29] interesting pod/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 30 01:27:54.774287 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:54.774173 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" podUID="03ccf6ef-c34a-4847-9791-1050db4d14d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 30 01:27:54.815284 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:54.815183 6221 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b219867f-db12-4b12-918a-24d4151ea2b7-openshift-service-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:27:54.815284 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:54.815222 6221 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b219867f-db12-4b12-918a-24d4151ea2b7-maas-api-tls\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:27:54.815284 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:54.815239 6221 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b219867f-db12-4b12-918a-24d4151ea2b7-metrics-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:27:54.815284 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:54.815253 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67cll\" (UniqueName: \"kubernetes.io/projected/b219867f-db12-4b12-918a-24d4151ea2b7-kube-api-access-67cll\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:27:54.918745 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:54.918713 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f19ba569-5fb5787844-wnqfr" Oct 30 01:27:55.003923 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:55.003891 6221 generic.go:356] "Generic (PLEG): container finished" podID="7b1bcfa7-d075-4aa9-aca3-afd74f76a125" containerID="9c3de723b98d993cf182daab6320b1c00f553be3b7bfaff09381f1f848aa8a9c" exitCode=0 Oct 30 01:27:55.004091 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:55.003943 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f19ba569-5fb5787844-wnqfr" Oct 30 01:27:55.004091 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:55.003978 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f19ba569-5fb5787844-wnqfr" event={"ID":"7b1bcfa7-d075-4aa9-aca3-afd74f76a125","Type":"ContainerDied","Data":"9c3de723b98d993cf182daab6320b1c00f553be3b7bfaff09381f1f848aa8a9c"} Oct 30 01:27:55.004091 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:55.004034 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f19ba569-5fb5787844-wnqfr" event={"ID":"7b1bcfa7-d075-4aa9-aca3-afd74f76a125","Type":"ContainerDied","Data":"63690a9d7545ee480e63c11c41f34d7f811d82866c128399b12fb602c7b4a1d2"} Oct 30 01:27:55.004091 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:55.004064 6221 scope.go:122] "RemoveContainer" containerID="9c3de723b98d993cf182daab6320b1c00f553be3b7bfaff09381f1f848aa8a9c" Oct 30 01:27:55.005473 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:55.005452 6221 generic.go:356] "Generic (PLEG): container finished" podID="b219867f-db12-4b12-918a-24d4151ea2b7" containerID="fbaca07c09731ac3325b0496786c2cc36c1120d56290a61b61f8788c0a7b39f7" exitCode=0 Oct 30 01:27:55.005611 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:55.005565 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9801220-f94447d96-7sv8l" event={"ID":"b219867f-db12-4b12-918a-24d4151ea2b7","Type":"ContainerDied","Data":"fbaca07c09731ac3325b0496786c2cc36c1120d56290a61b61f8788c0a7b39f7"} Oct 30 01:27:55.005611 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:55.005604 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9801220-f94447d96-7sv8l" event={"ID":"b219867f-db12-4b12-918a-24d4151ea2b7","Type":"ContainerDied","Data":"345e6318123bf562bc6431ec92c4ee06a9ed6acc9999c463965fed01c09481f3"} Oct 30 01:27:55.005733 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:55.005571 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d9801220-f94447d96-7sv8l" Oct 30 01:27:55.017922 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:55.017898 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b1bcfa7-d075-4aa9-aca3-afd74f76a125-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b1bcfa7-d075-4aa9-aca3-afd74f76a125-openshift-service-ca\") pod \"7b1bcfa7-d075-4aa9-aca3-afd74f76a125\" (UID: \"7b1bcfa7-d075-4aa9-aca3-afd74f76a125\") " Oct 30 01:27:55.018046 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:55.017984 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b1bcfa7-d075-4aa9-aca3-afd74f76a125-kube-api-access-qqtcs\" (UniqueName: \"kubernetes.io/projected/7b1bcfa7-d075-4aa9-aca3-afd74f76a125-kube-api-access-qqtcs\") pod \"7b1bcfa7-d075-4aa9-aca3-afd74f76a125\" (UID: \"7b1bcfa7-d075-4aa9-aca3-afd74f76a125\") " Oct 30 01:27:55.018046 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:55.018035 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b1bcfa7-d075-4aa9-aca3-afd74f76a125-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b1bcfa7-d075-4aa9-aca3-afd74f76a125-maas-api-tls\") pod \"7b1bcfa7-d075-4aa9-aca3-afd74f76a125\" (UID: \"7b1bcfa7-d075-4aa9-aca3-afd74f76a125\") " Oct 30 01:27:55.018158 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:55.018113 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b1bcfa7-d075-4aa9-aca3-afd74f76a125-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b1bcfa7-d075-4aa9-aca3-afd74f76a125-metrics-certs\") pod \"7b1bcfa7-d075-4aa9-aca3-afd74f76a125\" (UID: \"7b1bcfa7-d075-4aa9-aca3-afd74f76a125\") " Oct 30 01:27:55.018359 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:55.018336 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b1bcfa7-d075-4aa9-aca3-afd74f76a125-openshift-service-ca" pod "7b1bcfa7-d075-4aa9-aca3-afd74f76a125" (UID: "7b1bcfa7-d075-4aa9-aca3-afd74f76a125"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:27:55.018656 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:55.018636 6221 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b1bcfa7-d075-4aa9-aca3-afd74f76a125-openshift-service-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:27:55.019178 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:55.019156 6221 scope.go:122] "RemoveContainer" containerID="9c3de723b98d993cf182daab6320b1c00f553be3b7bfaff09381f1f848aa8a9c" Oct 30 01:27:55.019662 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:27:55.019628 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c3de723b98d993cf182daab6320b1c00f553be3b7bfaff09381f1f848aa8a9c\": container with ID starting with 9c3de723b98d993cf182daab6320b1c00f553be3b7bfaff09381f1f848aa8a9c not found: ID does not exist" containerID="9c3de723b98d993cf182daab6320b1c00f553be3b7bfaff09381f1f848aa8a9c" Oct 30 01:27:55.019768 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:55.019671 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c3de723b98d993cf182daab6320b1c00f553be3b7bfaff09381f1f848aa8a9c"} err="failed to get container status \"9c3de723b98d993cf182daab6320b1c00f553be3b7bfaff09381f1f848aa8a9c\": rpc error: code = NotFound desc = could not find container \"9c3de723b98d993cf182daab6320b1c00f553be3b7bfaff09381f1f848aa8a9c\": container with ID starting with 9c3de723b98d993cf182daab6320b1c00f553be3b7bfaff09381f1f848aa8a9c not found: ID does not exist" Oct 30 01:27:55.019768 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:55.019699 6221 scope.go:122] "RemoveContainer" containerID="fbaca07c09731ac3325b0496786c2cc36c1120d56290a61b61f8788c0a7b39f7" Oct 30 01:27:55.022162 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:55.021971 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b1bcfa7-d075-4aa9-aca3-afd74f76a125-metrics-certs" pod "7b1bcfa7-d075-4aa9-aca3-afd74f76a125" (UID: "7b1bcfa7-d075-4aa9-aca3-afd74f76a125"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:27:55.033978 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:55.033930 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b1bcfa7-d075-4aa9-aca3-afd74f76a125-kube-api-access-qqtcs" pod "7b1bcfa7-d075-4aa9-aca3-afd74f76a125" (UID: "7b1bcfa7-d075-4aa9-aca3-afd74f76a125"). InnerVolumeSpecName "kube-api-access-qqtcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:55.034085 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:55.033990 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b1bcfa7-d075-4aa9-aca3-afd74f76a125-maas-api-tls" pod "7b1bcfa7-d075-4aa9-aca3-afd74f76a125" (UID: "7b1bcfa7-d075-4aa9-aca3-afd74f76a125"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:27:55.037201 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:55.037155 6221 scope.go:122] "RemoveContainer" containerID="fbaca07c09731ac3325b0496786c2cc36c1120d56290a61b61f8788c0a7b39f7" Oct 30 01:27:55.037719 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:27:55.037697 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbaca07c09731ac3325b0496786c2cc36c1120d56290a61b61f8788c0a7b39f7\": container with ID starting with fbaca07c09731ac3325b0496786c2cc36c1120d56290a61b61f8788c0a7b39f7 not found: ID does not exist" containerID="fbaca07c09731ac3325b0496786c2cc36c1120d56290a61b61f8788c0a7b39f7" Oct 30 01:27:55.037842 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:55.037819 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbaca07c09731ac3325b0496786c2cc36c1120d56290a61b61f8788c0a7b39f7"} err="failed to get container status \"fbaca07c09731ac3325b0496786c2cc36c1120d56290a61b61f8788c0a7b39f7\": rpc error: code = NotFound desc = could not find container \"fbaca07c09731ac3325b0496786c2cc36c1120d56290a61b61f8788c0a7b39f7\": container with ID starting with fbaca07c09731ac3325b0496786c2cc36c1120d56290a61b61f8788c0a7b39f7 not found: ID does not exist" Oct 30 01:27:55.062893 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:55.062864 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d9801220-f94447d96-7sv8l"] Oct 30 01:27:55.068642 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:55.068608 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d9801220-f94447d96-7sv8l"] Oct 30 01:27:55.119941 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:55.119904 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqtcs\" (UniqueName: \"kubernetes.io/projected/7b1bcfa7-d075-4aa9-aca3-afd74f76a125-kube-api-access-qqtcs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:27:55.119941 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:55.119934 6221 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b1bcfa7-d075-4aa9-aca3-afd74f76a125-maas-api-tls\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:27:55.119941 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:55.119949 6221 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b1bcfa7-d075-4aa9-aca3-afd74f76a125-metrics-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:27:55.349645 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:55.349613 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f19ba569-5fb5787844-wnqfr"] Oct 30 01:27:55.351037 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:55.351013 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f19ba569-5fb5787844-wnqfr"] Oct 30 01:27:55.778114 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:55.778030 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" Oct 30 01:27:55.779082 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:55.779061 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" Oct 30 01:27:56.372752 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:56.372711 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz"] Oct 30 01:27:56.373335 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:56.373308 6221 cpu_manager.go:414] "containerMap: removing container" podUID="b219867f-db12-4b12-918a-24d4151ea2b7" containerName="maas-api" Oct 30 01:27:56.373335 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:56.373333 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b219867f-db12-4b12-918a-24d4151ea2b7" containerName="maas-api" Oct 30 01:27:56.373549 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:56.373365 6221 cpu_manager.go:414] "containerMap: removing container" podUID="2e4c7a82-6d00-4b93-bad8-f20425f04d40" containerName="payload-processing" Oct 30 01:27:56.373549 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:56.373383 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e4c7a82-6d00-4b93-bad8-f20425f04d40" containerName="payload-processing" Oct 30 01:27:56.373549 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:56.373396 6221 cpu_manager.go:414] "containerMap: removing container" podUID="f0d2d13d-4eb6-4a8a-8604-d893dbfe03df" containerName="payload-pre-processing" Oct 30 01:27:56.373549 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:56.373404 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0d2d13d-4eb6-4a8a-8604-d893dbfe03df" containerName="payload-pre-processing" Oct 30 01:27:56.373549 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:56.373415 6221 cpu_manager.go:414] "containerMap: removing container" podUID="7b1bcfa7-d075-4aa9-aca3-afd74f76a125" containerName="maas-api" Oct 30 01:27:56.373549 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:56.373422 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b1bcfa7-d075-4aa9-aca3-afd74f76a125" containerName="maas-api" Oct 30 01:27:56.373886 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:56.373561 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e4c7a82-6d00-4b93-bad8-f20425f04d40" containerName="payload-processing" Oct 30 01:27:56.373886 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:56.373577 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0d2d13d-4eb6-4a8a-8604-d893dbfe03df" containerName="payload-pre-processing" Oct 30 01:27:56.373886 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:56.373586 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b1bcfa7-d075-4aa9-aca3-afd74f76a125" containerName="maas-api" Oct 30 01:27:56.373886 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:56.373603 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="b219867f-db12-4b12-918a-24d4151ea2b7" containerName="maas-api" Oct 30 01:27:56.377340 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:56.377319 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" Oct 30 01:27:56.396653 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:56.396620 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-558327-openshift-default-dockercfg-rmc66\"" Oct 30 01:27:56.409422 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:56.409396 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz"] Oct 30 01:27:56.432293 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:56.432262 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-istio-data\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-zwshz\" (UID: \"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" Oct 30 01:27:56.432450 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:56.432323 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-workload-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-zwshz\" (UID: \"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" Oct 30 01:27:56.432504 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:56.432481 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-istio-envoy\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-zwshz\" (UID: \"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" Oct 30 01:27:56.432561 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:56.432522 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-istio-token\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-zwshz\" (UID: \"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" Oct 30 01:27:56.432628 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:56.432616 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-workload-certs\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-zwshz\" (UID: \"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" Oct 30 01:27:56.432666 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:56.432647 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgrx9\" (UniqueName: \"kubernetes.io/projected/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-kube-api-access-qgrx9\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-zwshz\" (UID: \"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" Oct 30 01:27:56.432700 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:56.432671 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-credential-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-zwshz\" (UID: \"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" Oct 30 01:27:56.432744 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:56.432694 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-istiod-ca-cert\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-zwshz\" (UID: \"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" Oct 30 01:27:56.432843 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:56.432823 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-istio-podinfo\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-zwshz\" (UID: \"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" Oct 30 01:27:56.535115 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:56.535078 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-istio-podinfo\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-zwshz\" (UID: \"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" Oct 30 01:27:56.535430 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:56.535406 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-istio-data\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-zwshz\" (UID: \"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" Oct 30 01:27:56.535595 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:56.535580 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-workload-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-zwshz\" (UID: \"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" Oct 30 01:27:56.535749 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:56.535736 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-istio-envoy\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-zwshz\" (UID: \"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" Oct 30 01:27:56.535883 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:56.535867 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-istio-token\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-zwshz\" (UID: \"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" Oct 30 01:27:56.536011 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:56.535993 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-workload-certs\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-zwshz\" (UID: \"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" Oct 30 01:27:56.536088 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:56.536040 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgrx9\" (UniqueName: \"kubernetes.io/projected/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-kube-api-access-qgrx9\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-zwshz\" (UID: \"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" Oct 30 01:27:56.536088 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:56.536078 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-credential-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-zwshz\" (UID: \"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" Oct 30 01:27:56.536191 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:56.536118 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-istiod-ca-cert\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-zwshz\" (UID: \"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" Oct 30 01:27:56.536991 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:56.536958 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-istiod-ca-cert\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-zwshz\" (UID: \"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" Oct 30 01:27:56.540498 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:56.539904 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-istio-data\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-zwshz\" (UID: \"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" Oct 30 01:27:56.540498 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:56.540202 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-workload-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-zwshz\" (UID: \"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" Oct 30 01:27:56.546599 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:56.546464 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-workload-certs\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-zwshz\" (UID: \"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" Oct 30 01:27:56.548625 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:56.548409 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-istio-envoy\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-zwshz\" (UID: \"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" Oct 30 01:27:56.548855 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:56.548832 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-credential-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-zwshz\" (UID: \"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" Oct 30 01:27:56.550757 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:56.550676 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-istio-podinfo\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-zwshz\" (UID: \"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" Oct 30 01:27:56.554922 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:56.554894 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b1bcfa7-d075-4aa9-aca3-afd74f76a125" path="/var/lib/kubelet/pods/7b1bcfa7-d075-4aa9-aca3-afd74f76a125/volumes" Oct 30 01:27:56.555372 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:56.555353 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b219867f-db12-4b12-918a-24d4151ea2b7" path="/var/lib/kubelet/pods/b219867f-db12-4b12-918a-24d4151ea2b7/volumes" Oct 30 01:27:56.562802 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:56.562774 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-istio-token\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-zwshz\" (UID: \"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" Oct 30 01:27:56.566210 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:56.566185 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgrx9\" (UniqueName: \"kubernetes.io/projected/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-kube-api-access-qgrx9\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-zwshz\" (UID: \"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" Oct 30 01:27:56.601380 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:56.601233 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2vsns_d332bbb4-c699-44e8-96ba-516d8ca03e95/ovn-acl-logging/0.log" Oct 30 01:27:56.603899 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:56.603882 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2vsns_d332bbb4-c699-44e8-96ba-516d8ca03e95/ovn-acl-logging/0.log" Oct 30 01:27:56.706121 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:56.706051 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-558327-openshift-default-dockercfg-rmc66\"" Oct 30 01:27:56.712119 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:56.712093 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" Oct 30 01:27:57.027076 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:57.026998 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz"] Oct 30 01:27:57.030063 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:27:57.030036 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod666e3fe7_fc6e_4aaa_9f63_f6223c6d07aa.slice/crio-c175d9c16a4645d44873c2543f9e0758d1746a068181d7a7150f2060d9aea532 WatchSource:0}: Error finding container c175d9c16a4645d44873c2543f9e0758d1746a068181d7a7150f2060d9aea532: Status 404 returned error can't find the container with id c175d9c16a4645d44873c2543f9e0758d1746a068181d7a7150f2060d9aea532 Oct 30 01:27:57.032154 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:57.032124 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:27:57.032212 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:57.032185 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:27:57.032244 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:57.032213 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:27:58.023267 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:58.023230 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" event={"ID":"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa","Type":"ContainerStarted","Data":"fc38a75f406fb4f803c813cf9685c74f1f5b1e2808a01db945239b2d6de1db4f"} Oct 30 01:27:58.023813 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:58.023287 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" event={"ID":"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa","Type":"ContainerStarted","Data":"c175d9c16a4645d44873c2543f9e0758d1746a068181d7a7150f2060d9aea532"} Oct 30 01:27:58.052257 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:58.052232 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5b724cd8-6fd679fd8-phz5k"] Oct 30 01:27:58.083621 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:58.083577 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" podStartSLOduration=2.083563682 podStartE2EDuration="2.083563682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:27:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:27:58.082282875 +0000 UTC m=+601.660570210" watchObservedRunningTime="2026-10-30 01:27:58.083563682 +0000 UTC m=+601.661851097" Oct 30 01:27:58.443808 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:58.443777 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:27:58.712387 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:58.712296 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" Oct 30 01:27:58.717876 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:58.717841 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" Oct 30 01:27:58.778888 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:58.778856 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5b724cd8-74fd7b978-7htgh"] Oct 30 01:27:58.779122 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:58.779094 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b724cd8-74fd7b978-7htgh" podUID="e4e5b900-9733-4b6e-9d10-8284175d21f5" containerName="maas-api" containerID="cri-o://f42c0b7a616e50554f67c98b89111125f2d210bd1a1299a8bd81b2f3d360299a" gracePeriod=30 Oct 30 01:27:58.861420 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:58.861369 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b724cd8-74fd7b978-7htgh" podUID="e4e5b900-9733-4b6e-9d10-8284175d21f5" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.45:8443/health\": dial tcp 10.134.0.45:8443: connect: connection refused" Oct 30 01:27:59.019187 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:59.019165 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b724cd8-74fd7b978-7htgh" Oct 30 01:27:59.028458 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:59.028432 6221 generic.go:356] "Generic (PLEG): container finished" podID="e4e5b900-9733-4b6e-9d10-8284175d21f5" containerID="f42c0b7a616e50554f67c98b89111125f2d210bd1a1299a8bd81b2f3d360299a" exitCode=0 Oct 30 01:27:59.028805 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:59.028465 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b724cd8-74fd7b978-7htgh" Oct 30 01:27:59.028805 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:59.028492 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b724cd8-74fd7b978-7htgh" event={"ID":"e4e5b900-9733-4b6e-9d10-8284175d21f5","Type":"ContainerDied","Data":"f42c0b7a616e50554f67c98b89111125f2d210bd1a1299a8bd81b2f3d360299a"} Oct 30 01:27:59.028805 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:59.028547 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-5b724cd8-74fd7b978-7htgh" event={"ID":"e4e5b900-9733-4b6e-9d10-8284175d21f5","Type":"ContainerDied","Data":"620a11864e74a82d39c6601808b0a6784747318fadf87d72f8053520626cbeb7"} Oct 30 01:27:59.028805 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:59.028574 6221 scope.go:122] "RemoveContainer" containerID="f42c0b7a616e50554f67c98b89111125f2d210bd1a1299a8bd81b2f3d360299a" Oct 30 01:27:59.028971 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:59.028844 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" Oct 30 01:27:59.029988 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:59.029967 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" Oct 30 01:27:59.039624 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:59.039606 6221 scope.go:122] "RemoveContainer" containerID="f42c0b7a616e50554f67c98b89111125f2d210bd1a1299a8bd81b2f3d360299a" Oct 30 01:27:59.039941 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:27:59.039919 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f42c0b7a616e50554f67c98b89111125f2d210bd1a1299a8bd81b2f3d360299a\": container with ID starting with f42c0b7a616e50554f67c98b89111125f2d210bd1a1299a8bd81b2f3d360299a not found: ID does not exist" containerID="f42c0b7a616e50554f67c98b89111125f2d210bd1a1299a8bd81b2f3d360299a" Oct 30 01:27:59.040022 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:59.039946 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f42c0b7a616e50554f67c98b89111125f2d210bd1a1299a8bd81b2f3d360299a"} err="failed to get container status \"f42c0b7a616e50554f67c98b89111125f2d210bd1a1299a8bd81b2f3d360299a\": rpc error: code = NotFound desc = could not find container \"f42c0b7a616e50554f67c98b89111125f2d210bd1a1299a8bd81b2f3d360299a\": container with ID starting with f42c0b7a616e50554f67c98b89111125f2d210bd1a1299a8bd81b2f3d360299a not found: ID does not exist" Oct 30 01:27:59.120501 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:59.120468 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f29772b9-c9bbdbc8f-tkpwz"] Oct 30 01:27:59.120696 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:59.120665 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f29772b9-c9bbdbc8f-tkpwz" podUID="58489c7b-813d-488b-936f-0772735cb950" containerName="maas-api" containerID="cri-o://55b292f557a6794f88ed73cf0e7aed5a8231e61a9b9d213675711a41ae88cfc9" gracePeriod=30 Oct 30 01:27:59.168703 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:59.168671 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4e5b900-9733-4b6e-9d10-8284175d21f5-kube-api-access-rqfd2\" (UniqueName: \"kubernetes.io/projected/e4e5b900-9733-4b6e-9d10-8284175d21f5-kube-api-access-rqfd2\") pod \"e4e5b900-9733-4b6e-9d10-8284175d21f5\" (UID: \"e4e5b900-9733-4b6e-9d10-8284175d21f5\") " Oct 30 01:27:59.168880 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:59.168775 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4e5b900-9733-4b6e-9d10-8284175d21f5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4e5b900-9733-4b6e-9d10-8284175d21f5-maas-api-tls\") pod \"e4e5b900-9733-4b6e-9d10-8284175d21f5\" (UID: \"e4e5b900-9733-4b6e-9d10-8284175d21f5\") " Oct 30 01:27:59.168880 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:59.168868 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4e5b900-9733-4b6e-9d10-8284175d21f5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4e5b900-9733-4b6e-9d10-8284175d21f5-openshift-service-ca\") pod \"e4e5b900-9733-4b6e-9d10-8284175d21f5\" (UID: \"e4e5b900-9733-4b6e-9d10-8284175d21f5\") " Oct 30 01:27:59.168997 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:59.168901 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4e5b900-9733-4b6e-9d10-8284175d21f5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4e5b900-9733-4b6e-9d10-8284175d21f5-metrics-certs\") pod \"e4e5b900-9733-4b6e-9d10-8284175d21f5\" (UID: \"e4e5b900-9733-4b6e-9d10-8284175d21f5\") " Oct 30 01:27:59.169636 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:59.169258 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4e5b900-9733-4b6e-9d10-8284175d21f5-openshift-service-ca" pod "e4e5b900-9733-4b6e-9d10-8284175d21f5" (UID: "e4e5b900-9733-4b6e-9d10-8284175d21f5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:27:59.171300 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:59.171277 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4e5b900-9733-4b6e-9d10-8284175d21f5-metrics-certs" pod "e4e5b900-9733-4b6e-9d10-8284175d21f5" (UID: "e4e5b900-9733-4b6e-9d10-8284175d21f5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:27:59.171466 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:59.171446 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4e5b900-9733-4b6e-9d10-8284175d21f5-kube-api-access-rqfd2" pod "e4e5b900-9733-4b6e-9d10-8284175d21f5" (UID: "e4e5b900-9733-4b6e-9d10-8284175d21f5"). InnerVolumeSpecName "kube-api-access-rqfd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:59.171590 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:59.171569 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4e5b900-9733-4b6e-9d10-8284175d21f5-maas-api-tls" pod "e4e5b900-9733-4b6e-9d10-8284175d21f5" (UID: "e4e5b900-9733-4b6e-9d10-8284175d21f5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:27:59.270275 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:59.270243 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqfd2\" (UniqueName: \"kubernetes.io/projected/e4e5b900-9733-4b6e-9d10-8284175d21f5-kube-api-access-rqfd2\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:27:59.270275 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:59.270271 6221 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4e5b900-9733-4b6e-9d10-8284175d21f5-maas-api-tls\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:27:59.270275 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:59.270280 6221 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4e5b900-9733-4b6e-9d10-8284175d21f5-openshift-service-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:27:59.270557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:59.270289 6221 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4e5b900-9733-4b6e-9d10-8284175d21f5-metrics-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:27:59.381623 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:59.381591 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5b724cd8-74fd7b978-7htgh"] Oct 30 01:27:59.386671 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:59.386650 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f29772b9-c9bbdbc8f-tkpwz" Oct 30 01:27:59.389961 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:59.389940 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-5b724cd8-74fd7b978-7htgh"] Oct 30 01:27:59.471756 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:59.471726 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58489c7b-813d-488b-936f-0772735cb950-kube-api-access-mdmfh\" (UniqueName: \"kubernetes.io/projected/58489c7b-813d-488b-936f-0772735cb950-kube-api-access-mdmfh\") pod \"58489c7b-813d-488b-936f-0772735cb950\" (UID: \"58489c7b-813d-488b-936f-0772735cb950\") " Oct 30 01:27:59.471918 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:59.471791 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58489c7b-813d-488b-936f-0772735cb950-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58489c7b-813d-488b-936f-0772735cb950-openshift-service-ca\") pod \"58489c7b-813d-488b-936f-0772735cb950\" (UID: \"58489c7b-813d-488b-936f-0772735cb950\") " Oct 30 01:27:59.471918 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:59.471842 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58489c7b-813d-488b-936f-0772735cb950-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58489c7b-813d-488b-936f-0772735cb950-maas-api-tls\") pod \"58489c7b-813d-488b-936f-0772735cb950\" (UID: \"58489c7b-813d-488b-936f-0772735cb950\") " Oct 30 01:27:59.471918 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:59.471868 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58489c7b-813d-488b-936f-0772735cb950-metrics-certs\" (UniqueName: \"kubernetes.io/secret/58489c7b-813d-488b-936f-0772735cb950-metrics-certs\") pod \"58489c7b-813d-488b-936f-0772735cb950\" (UID: \"58489c7b-813d-488b-936f-0772735cb950\") " Oct 30 01:27:59.472195 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:59.472167 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58489c7b-813d-488b-936f-0772735cb950-openshift-service-ca" pod "58489c7b-813d-488b-936f-0772735cb950" (UID: "58489c7b-813d-488b-936f-0772735cb950"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:27:59.473905 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:59.473877 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58489c7b-813d-488b-936f-0772735cb950-metrics-certs" pod "58489c7b-813d-488b-936f-0772735cb950" (UID: "58489c7b-813d-488b-936f-0772735cb950"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:27:59.474325 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:59.474299 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58489c7b-813d-488b-936f-0772735cb950-maas-api-tls" pod "58489c7b-813d-488b-936f-0772735cb950" (UID: "58489c7b-813d-488b-936f-0772735cb950"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:27:59.474382 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:59.474303 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58489c7b-813d-488b-936f-0772735cb950-kube-api-access-mdmfh" pod "58489c7b-813d-488b-936f-0772735cb950" (UID: "58489c7b-813d-488b-936f-0772735cb950"). InnerVolumeSpecName "kube-api-access-mdmfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:27:59.566981 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:59.566937 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p"] Oct 30 01:27:59.567173 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:59.567151 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" podUID="ec361aac-99e4-4f3c-9c0a-33411f8f74e3" containerName="istio-proxy" containerID="cri-o://e2e40b5f725217a45cd267db9bba839c6b2665ca2fa0b45028aabd990910324b" gracePeriod=30 Oct 30 01:27:59.573466 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:59.573439 6221 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58489c7b-813d-488b-936f-0772735cb950-maas-api-tls\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:27:59.573466 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:59.573463 6221 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58489c7b-813d-488b-936f-0772735cb950-metrics-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:27:59.573676 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:59.573472 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdmfh\" (UniqueName: \"kubernetes.io/projected/58489c7b-813d-488b-936f-0772735cb950-kube-api-access-mdmfh\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:27:59.573676 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:27:59.573481 6221 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58489c7b-813d-488b-936f-0772735cb950-openshift-service-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:28:00.036466 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:00.036427 6221 generic.go:356] "Generic (PLEG): container finished" podID="58489c7b-813d-488b-936f-0772735cb950" containerID="55b292f557a6794f88ed73cf0e7aed5a8231e61a9b9d213675711a41ae88cfc9" exitCode=0 Oct 30 01:28:00.037004 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:00.036491 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f29772b9-c9bbdbc8f-tkpwz" Oct 30 01:28:00.037004 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:00.036523 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f29772b9-c9bbdbc8f-tkpwz" event={"ID":"58489c7b-813d-488b-936f-0772735cb950","Type":"ContainerDied","Data":"55b292f557a6794f88ed73cf0e7aed5a8231e61a9b9d213675711a41ae88cfc9"} Oct 30 01:28:00.037004 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:00.036648 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f29772b9-c9bbdbc8f-tkpwz" event={"ID":"58489c7b-813d-488b-936f-0772735cb950","Type":"ContainerDied","Data":"5a5991559ece16abfa2b59e74348d170911b4063013ee031f2f8bb2e2e03ec77"} Oct 30 01:28:00.037004 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:00.036686 6221 scope.go:122] "RemoveContainer" containerID="55b292f557a6794f88ed73cf0e7aed5a8231e61a9b9d213675711a41ae88cfc9" Oct 30 01:28:00.048741 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:00.048718 6221 scope.go:122] "RemoveContainer" containerID="55b292f557a6794f88ed73cf0e7aed5a8231e61a9b9d213675711a41ae88cfc9" Oct 30 01:28:00.049099 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:28:00.049077 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55b292f557a6794f88ed73cf0e7aed5a8231e61a9b9d213675711a41ae88cfc9\": container with ID starting with 55b292f557a6794f88ed73cf0e7aed5a8231e61a9b9d213675711a41ae88cfc9 not found: ID does not exist" containerID="55b292f557a6794f88ed73cf0e7aed5a8231e61a9b9d213675711a41ae88cfc9" Oct 30 01:28:00.049213 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:00.049106 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55b292f557a6794f88ed73cf0e7aed5a8231e61a9b9d213675711a41ae88cfc9"} err="failed to get container status \"55b292f557a6794f88ed73cf0e7aed5a8231e61a9b9d213675711a41ae88cfc9\": rpc error: code = NotFound desc = could not find container \"55b292f557a6794f88ed73cf0e7aed5a8231e61a9b9d213675711a41ae88cfc9\": container with ID starting with 55b292f557a6794f88ed73cf0e7aed5a8231e61a9b9d213675711a41ae88cfc9 not found: ID does not exist" Oct 30 01:28:00.087523 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:00.087463 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f29772b9-c9bbdbc8f-tkpwz"] Oct 30 01:28:00.089306 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:00.089278 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f29772b9-c9bbdbc8f-tkpwz"] Oct 30 01:28:00.545873 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:00.545844 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58489c7b-813d-488b-936f-0772735cb950" path="/var/lib/kubelet/pods/58489c7b-813d-488b-936f-0772735cb950/volumes" Oct 30 01:28:00.546259 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:00.546247 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4e5b900-9733-4b6e-9d10-8284175d21f5" path="/var/lib/kubelet/pods/e4e5b900-9733-4b6e-9d10-8284175d21f5/volumes" Oct 30 01:28:02.519098 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:02.517938 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-558327-bd6c69d5b-scg4b"] Oct 30 01:28:02.519098 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:02.518909 6221 cpu_manager.go:414] "containerMap: removing container" podUID="e4e5b900-9733-4b6e-9d10-8284175d21f5" containerName="maas-api" Oct 30 01:28:02.519098 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:02.519000 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4e5b900-9733-4b6e-9d10-8284175d21f5" containerName="maas-api" Oct 30 01:28:02.519098 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:02.519065 6221 cpu_manager.go:414] "containerMap: removing container" podUID="58489c7b-813d-488b-936f-0772735cb950" containerName="maas-api" Oct 30 01:28:02.519098 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:02.519074 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58489c7b-813d-488b-936f-0772735cb950" containerName="maas-api" Oct 30 01:28:02.519907 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:02.519467 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4e5b900-9733-4b6e-9d10-8284175d21f5" containerName="maas-api" Oct 30 01:28:02.519907 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:02.519490 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="58489c7b-813d-488b-936f-0772735cb950" containerName="maas-api" Oct 30 01:28:02.525188 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:02.525153 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-558327-bd6c69d5b-scg4b" Oct 30 01:28:02.555478 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:02.555449 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-558327\"" Oct 30 01:28:02.557178 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:02.557155 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-558327-bd6c69d5b-scg4b"] Oct 30 01:28:02.626987 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:02.626933 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/125fc809-a755-460c-8605-363489a85b49-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-558327-bd6c69d5b-scg4b\" (UID: \"125fc809-a755-460c-8605-363489a85b49\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-558327-bd6c69d5b-scg4b" Oct 30 01:28:02.627184 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:02.627056 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7zr8\" (UniqueName: \"kubernetes.io/projected/125fc809-a755-460c-8605-363489a85b49-kube-api-access-h7zr8\") pod \"payload-processing-e2e-worker-w3-558327-bd6c69d5b-scg4b\" (UID: \"125fc809-a755-460c-8605-363489a85b49\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-558327-bd6c69d5b-scg4b" Oct 30 01:28:02.727885 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:02.727849 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/125fc809-a755-460c-8605-363489a85b49-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-558327-bd6c69d5b-scg4b\" (UID: \"125fc809-a755-460c-8605-363489a85b49\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-558327-bd6c69d5b-scg4b" Oct 30 01:28:02.728055 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:02.727947 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7zr8\" (UniqueName: \"kubernetes.io/projected/125fc809-a755-460c-8605-363489a85b49-kube-api-access-h7zr8\") pod \"payload-processing-e2e-worker-w3-558327-bd6c69d5b-scg4b\" (UID: \"125fc809-a755-460c-8605-363489a85b49\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-558327-bd6c69d5b-scg4b" Oct 30 01:28:02.728569 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:02.728549 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/125fc809-a755-460c-8605-363489a85b49-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-558327-bd6c69d5b-scg4b\" (UID: \"125fc809-a755-460c-8605-363489a85b49\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-558327-bd6c69d5b-scg4b" Oct 30 01:28:02.747349 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:02.747315 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7zr8\" (UniqueName: \"kubernetes.io/projected/125fc809-a755-460c-8605-363489a85b49-kube-api-access-h7zr8\") pod \"payload-processing-e2e-worker-w3-558327-bd6c69d5b-scg4b\" (UID: \"125fc809-a755-460c-8605-363489a85b49\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-558327-bd6c69d5b-scg4b" Oct 30 01:28:02.839737 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:02.839700 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-558327-bd6c69d5b-scg4b" Oct 30 01:28:03.052854 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:03.052816 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-5b724cd8-6fd679fd8-phz5k" podUID="313c4f85-855a-4e1a-b3d3-ae2d4e038bce" containerName="payload-processing" containerID="cri-o://f20a8cfd3a1a02aeaa5e646f9f88ec68c63a31dd5a1ca997bd5d918b0c7de595" gracePeriod=25 Oct 30 01:28:03.138183 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:03.138089 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-558327-bd6c69d5b-scg4b"] Oct 30 01:28:03.143402 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:28:03.143369 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod125fc809_a755_460c_8605_363489a85b49.slice/crio-b8d7fc6136679a97ac007bb424acda23a6d0053c4572d4aa74aa29777c789416 WatchSource:0}: Error finding container b8d7fc6136679a97ac007bb424acda23a6d0053c4572d4aa74aa29777c789416: Status 404 returned error can't find the container with id b8d7fc6136679a97ac007bb424acda23a6d0053c4572d4aa74aa29777c789416 Oct 30 01:28:03.293396 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:03.293361 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5b724cd8-6fd679fd8-phz5k" Oct 30 01:28:03.436289 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:03.436203 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/313c4f85-855a-4e1a-b3d3-ae2d4e038bce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/313c4f85-855a-4e1a-b3d3-ae2d4e038bce-plugins-config-volume\") pod \"313c4f85-855a-4e1a-b3d3-ae2d4e038bce\" (UID: \"313c4f85-855a-4e1a-b3d3-ae2d4e038bce\") " Oct 30 01:28:03.436289 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:03.436240 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/313c4f85-855a-4e1a-b3d3-ae2d4e038bce-kube-api-access-hbnpz\" (UniqueName: \"kubernetes.io/projected/313c4f85-855a-4e1a-b3d3-ae2d4e038bce-kube-api-access-hbnpz\") pod \"313c4f85-855a-4e1a-b3d3-ae2d4e038bce\" (UID: \"313c4f85-855a-4e1a-b3d3-ae2d4e038bce\") " Oct 30 01:28:03.436680 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:03.436654 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/313c4f85-855a-4e1a-b3d3-ae2d4e038bce-plugins-config-volume" pod "313c4f85-855a-4e1a-b3d3-ae2d4e038bce" (UID: "313c4f85-855a-4e1a-b3d3-ae2d4e038bce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:28:03.438384 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:03.438360 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/313c4f85-855a-4e1a-b3d3-ae2d4e038bce-kube-api-access-hbnpz" pod "313c4f85-855a-4e1a-b3d3-ae2d4e038bce" (UID: "313c4f85-855a-4e1a-b3d3-ae2d4e038bce"). InnerVolumeSpecName "kube-api-access-hbnpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:03.537570 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:03.537524 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/313c4f85-855a-4e1a-b3d3-ae2d4e038bce-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:28:03.537570 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:03.537567 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbnpz\" (UniqueName: \"kubernetes.io/projected/313c4f85-855a-4e1a-b3d3-ae2d4e038bce-kube-api-access-hbnpz\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:28:03.588585 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:03.588550 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-558327-7945456798-frsjf"] Oct 30 01:28:03.588974 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:03.588959 6221 cpu_manager.go:414] "containerMap: removing container" podUID="313c4f85-855a-4e1a-b3d3-ae2d4e038bce" containerName="payload-processing" Oct 30 01:28:03.589014 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:03.588977 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="313c4f85-855a-4e1a-b3d3-ae2d4e038bce" containerName="payload-processing" Oct 30 01:28:03.589073 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:03.589063 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="313c4f85-855a-4e1a-b3d3-ae2d4e038bce" containerName="payload-processing" Oct 30 01:28:03.592160 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:03.592143 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-558327-7945456798-frsjf" Oct 30 01:28:03.607310 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:03.607275 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-558327-serving-cert\"" Oct 30 01:28:03.609285 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:03.609258 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-558327-7945456798-frsjf"] Oct 30 01:28:03.740224 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:03.740143 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cbfa236-b599-4625-bf0f-8d5d3e47398e-metrics-certs\") pod \"maas-api-e2e-worker-w3-558327-7945456798-frsjf\" (UID: \"4cbfa236-b599-4625-bf0f-8d5d3e47398e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-558327-7945456798-frsjf" Oct 30 01:28:03.740224 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:03.740194 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cbfa236-b599-4625-bf0f-8d5d3e47398e-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-558327-7945456798-frsjf\" (UID: \"4cbfa236-b599-4625-bf0f-8d5d3e47398e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-558327-7945456798-frsjf" Oct 30 01:28:03.740407 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:03.740274 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crtj8\" (UniqueName: \"kubernetes.io/projected/4cbfa236-b599-4625-bf0f-8d5d3e47398e-kube-api-access-crtj8\") pod \"maas-api-e2e-worker-w3-558327-7945456798-frsjf\" (UID: \"4cbfa236-b599-4625-bf0f-8d5d3e47398e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-558327-7945456798-frsjf" Oct 30 01:28:03.740407 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:03.740316 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cbfa236-b599-4625-bf0f-8d5d3e47398e-maas-api-tls\") pod \"maas-api-e2e-worker-w3-558327-7945456798-frsjf\" (UID: \"4cbfa236-b599-4625-bf0f-8d5d3e47398e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-558327-7945456798-frsjf" Oct 30 01:28:03.841676 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:03.841636 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cbfa236-b599-4625-bf0f-8d5d3e47398e-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-558327-7945456798-frsjf\" (UID: \"4cbfa236-b599-4625-bf0f-8d5d3e47398e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-558327-7945456798-frsjf" Oct 30 01:28:03.841881 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:03.841683 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crtj8\" (UniqueName: \"kubernetes.io/projected/4cbfa236-b599-4625-bf0f-8d5d3e47398e-kube-api-access-crtj8\") pod \"maas-api-e2e-worker-w3-558327-7945456798-frsjf\" (UID: \"4cbfa236-b599-4625-bf0f-8d5d3e47398e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-558327-7945456798-frsjf" Oct 30 01:28:03.841881 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:03.841712 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cbfa236-b599-4625-bf0f-8d5d3e47398e-maas-api-tls\") pod \"maas-api-e2e-worker-w3-558327-7945456798-frsjf\" (UID: \"4cbfa236-b599-4625-bf0f-8d5d3e47398e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-558327-7945456798-frsjf" Oct 30 01:28:03.841881 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:03.841807 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cbfa236-b599-4625-bf0f-8d5d3e47398e-metrics-certs\") pod \"maas-api-e2e-worker-w3-558327-7945456798-frsjf\" (UID: \"4cbfa236-b599-4625-bf0f-8d5d3e47398e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-558327-7945456798-frsjf" Oct 30 01:28:03.842299 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:03.842276 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cbfa236-b599-4625-bf0f-8d5d3e47398e-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-558327-7945456798-frsjf\" (UID: \"4cbfa236-b599-4625-bf0f-8d5d3e47398e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-558327-7945456798-frsjf" Oct 30 01:28:03.844198 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:03.844164 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cbfa236-b599-4625-bf0f-8d5d3e47398e-metrics-certs\") pod \"maas-api-e2e-worker-w3-558327-7945456798-frsjf\" (UID: \"4cbfa236-b599-4625-bf0f-8d5d3e47398e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-558327-7945456798-frsjf" Oct 30 01:28:03.844375 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:03.844350 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cbfa236-b599-4625-bf0f-8d5d3e47398e-maas-api-tls\") pod \"maas-api-e2e-worker-w3-558327-7945456798-frsjf\" (UID: \"4cbfa236-b599-4625-bf0f-8d5d3e47398e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-558327-7945456798-frsjf" Oct 30 01:28:03.861462 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:03.861439 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crtj8\" (UniqueName: \"kubernetes.io/projected/4cbfa236-b599-4625-bf0f-8d5d3e47398e-kube-api-access-crtj8\") pod \"maas-api-e2e-worker-w3-558327-7945456798-frsjf\" (UID: \"4cbfa236-b599-4625-bf0f-8d5d3e47398e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-558327-7945456798-frsjf" Oct 30 01:28:03.905473 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:03.905435 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-558327-7945456798-frsjf" Oct 30 01:28:04.063001 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:04.062966 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-558327-bd6c69d5b-scg4b" event={"ID":"125fc809-a755-460c-8605-363489a85b49","Type":"ContainerStarted","Data":"b8652bab3d6525ebae066d2815fac5993d0c3e10a7347e93eeda9a901338de46"} Oct 30 01:28:04.063186 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:04.063026 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-558327-bd6c69d5b-scg4b" event={"ID":"125fc809-a755-460c-8605-363489a85b49","Type":"ContainerStarted","Data":"b8d7fc6136679a97ac007bb424acda23a6d0053c4572d4aa74aa29777c789416"} Oct 30 01:28:04.063186 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:04.063148 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-558327-bd6c69d5b-scg4b" Oct 30 01:28:04.064595 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:04.064568 6221 generic.go:356] "Generic (PLEG): container finished" podID="313c4f85-855a-4e1a-b3d3-ae2d4e038bce" containerID="f20a8cfd3a1a02aeaa5e646f9f88ec68c63a31dd5a1ca997bd5d918b0c7de595" exitCode=0 Oct 30 01:28:04.064754 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:04.064608 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5b724cd8-6fd679fd8-phz5k" Oct 30 01:28:04.064754 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:04.064691 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5b724cd8-6fd679fd8-phz5k" event={"ID":"313c4f85-855a-4e1a-b3d3-ae2d4e038bce","Type":"ContainerDied","Data":"f20a8cfd3a1a02aeaa5e646f9f88ec68c63a31dd5a1ca997bd5d918b0c7de595"} Oct 30 01:28:04.064754 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:04.064721 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5b724cd8-6fd679fd8-phz5k" event={"ID":"313c4f85-855a-4e1a-b3d3-ae2d4e038bce","Type":"ContainerDied","Data":"c51e1fc3adc970df84e0f8576829f9f4db26d09bb10654c8ac50f70a5fee80b6"} Oct 30 01:28:04.064754 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:04.064740 6221 scope.go:122] "RemoveContainer" containerID="f20a8cfd3a1a02aeaa5e646f9f88ec68c63a31dd5a1ca997bd5d918b0c7de595" Oct 30 01:28:04.075108 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:04.075090 6221 scope.go:122] "RemoveContainer" containerID="f20a8cfd3a1a02aeaa5e646f9f88ec68c63a31dd5a1ca997bd5d918b0c7de595" Oct 30 01:28:04.075426 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:28:04.075407 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f20a8cfd3a1a02aeaa5e646f9f88ec68c63a31dd5a1ca997bd5d918b0c7de595\": container with ID starting with f20a8cfd3a1a02aeaa5e646f9f88ec68c63a31dd5a1ca997bd5d918b0c7de595 not found: ID does not exist" containerID="f20a8cfd3a1a02aeaa5e646f9f88ec68c63a31dd5a1ca997bd5d918b0c7de595" Oct 30 01:28:04.075492 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:04.075434 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f20a8cfd3a1a02aeaa5e646f9f88ec68c63a31dd5a1ca997bd5d918b0c7de595"} err="failed to get container status \"f20a8cfd3a1a02aeaa5e646f9f88ec68c63a31dd5a1ca997bd5d918b0c7de595\": rpc error: code = NotFound desc = could not find container \"f20a8cfd3a1a02aeaa5e646f9f88ec68c63a31dd5a1ca997bd5d918b0c7de595\": container with ID starting with f20a8cfd3a1a02aeaa5e646f9f88ec68c63a31dd5a1ca997bd5d918b0c7de595 not found: ID does not exist" Oct 30 01:28:04.102816 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:04.102761 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-558327-bd6c69d5b-scg4b" podStartSLOduration=2.102745834 podStartE2EDuration="2.102745834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:04.102028098 +0000 UTC m=+607.680315433" watchObservedRunningTime="2026-10-30 01:28:04.102745834 +0000 UTC m=+607.681033169" Oct 30 01:28:04.137105 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:04.137071 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5b724cd8-6fd679fd8-phz5k"] Oct 30 01:28:04.141307 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:04.141276 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5b724cd8-6fd679fd8-phz5k"] Oct 30 01:28:04.234472 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:04.234423 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-558327-7945456798-frsjf"] Oct 30 01:28:04.241180 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:28:04.241114 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cbfa236_b599_4625_bf0f_8d5d3e47398e.slice/crio-6bd7e065505353fd5a24bd32e76279791c80bbf1cfb51f39ce4a7ba177e8835b WatchSource:0}: Error finding container 6bd7e065505353fd5a24bd32e76279791c80bbf1cfb51f39ce4a7ba177e8835b: Status 404 returned error can't find the container with id 6bd7e065505353fd5a24bd32e76279791c80bbf1cfb51f39ce4a7ba177e8835b Oct 30 01:28:04.546769 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:04.546734 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="313c4f85-855a-4e1a-b3d3-ae2d4e038bce" path="/var/lib/kubelet/pods/313c4f85-855a-4e1a-b3d3-ae2d4e038bce/volumes" Oct 30 01:28:04.859354 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:04.859330 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" Oct 30 01:28:04.963189 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:04.963152 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-istiod-ca-cert\") pod \"ec361aac-99e4-4f3c-9c0a-33411f8f74e3\" (UID: \"ec361aac-99e4-4f3c-9c0a-33411f8f74e3\") " Oct 30 01:28:04.963374 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:04.963199 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-istio-token\" (UniqueName: \"kubernetes.io/projected/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-istio-token\") pod \"ec361aac-99e4-4f3c-9c0a-33411f8f74e3\" (UID: \"ec361aac-99e4-4f3c-9c0a-33411f8f74e3\") " Oct 30 01:28:04.963374 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:04.963247 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-istio-data\") pod \"ec361aac-99e4-4f3c-9c0a-33411f8f74e3\" (UID: \"ec361aac-99e4-4f3c-9c0a-33411f8f74e3\") " Oct 30 01:28:04.963374 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:04.963281 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-kube-api-access-gkffw\" (UniqueName: \"kubernetes.io/projected/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-kube-api-access-gkffw\") pod \"ec361aac-99e4-4f3c-9c0a-33411f8f74e3\" (UID: \"ec361aac-99e4-4f3c-9c0a-33411f8f74e3\") " Oct 30 01:28:04.963374 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:04.963322 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-workload-socket\") pod \"ec361aac-99e4-4f3c-9c0a-33411f8f74e3\" (UID: \"ec361aac-99e4-4f3c-9c0a-33411f8f74e3\") " Oct 30 01:28:04.963374 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:04.963350 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-workload-certs\") pod \"ec361aac-99e4-4f3c-9c0a-33411f8f74e3\" (UID: \"ec361aac-99e4-4f3c-9c0a-33411f8f74e3\") " Oct 30 01:28:04.963686 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:04.963389 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-credential-socket\") pod \"ec361aac-99e4-4f3c-9c0a-33411f8f74e3\" (UID: \"ec361aac-99e4-4f3c-9c0a-33411f8f74e3\") " Oct 30 01:28:04.963686 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:04.963418 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-istio-envoy\") pod \"ec361aac-99e4-4f3c-9c0a-33411f8f74e3\" (UID: \"ec361aac-99e4-4f3c-9c0a-33411f8f74e3\") " Oct 30 01:28:04.963686 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:04.963493 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-istio-podinfo\") pod \"ec361aac-99e4-4f3c-9c0a-33411f8f74e3\" (UID: \"ec361aac-99e4-4f3c-9c0a-33411f8f74e3\") " Oct 30 01:28:04.963686 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:04.963613 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-istio-data" pod "ec361aac-99e4-4f3c-9c0a-33411f8f74e3" (UID: "ec361aac-99e4-4f3c-9c0a-33411f8f74e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:04.963865 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:04.963713 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-workload-certs" pod "ec361aac-99e4-4f3c-9c0a-33411f8f74e3" (UID: "ec361aac-99e4-4f3c-9c0a-33411f8f74e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:04.963865 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:04.963796 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-istiod-ca-cert" pod "ec361aac-99e4-4f3c-9c0a-33411f8f74e3" (UID: "ec361aac-99e4-4f3c-9c0a-33411f8f74e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:28:04.963962 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:04.963865 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-workload-socket" pod "ec361aac-99e4-4f3c-9c0a-33411f8f74e3" (UID: "ec361aac-99e4-4f3c-9c0a-33411f8f74e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:04.963962 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:04.963912 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-credential-socket" pod "ec361aac-99e4-4f3c-9c0a-33411f8f74e3" (UID: "ec361aac-99e4-4f3c-9c0a-33411f8f74e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:04.964126 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:04.964109 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:28:04.964191 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:04.964131 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:28:04.964191 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:04.964145 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:28:04.964191 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:04.964154 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:28:04.964191 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:04.964163 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:28:04.965713 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:04.965690 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-istio-envoy" pod "ec361aac-99e4-4f3c-9c0a-33411f8f74e3" (UID: "ec361aac-99e4-4f3c-9c0a-33411f8f74e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:04.965820 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:04.965737 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-istio-podinfo" pod "ec361aac-99e4-4f3c-9c0a-33411f8f74e3" (UID: "ec361aac-99e4-4f3c-9c0a-33411f8f74e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:28:04.965968 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:04.965952 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-kube-api-access-gkffw" pod "ec361aac-99e4-4f3c-9c0a-33411f8f74e3" (UID: "ec361aac-99e4-4f3c-9c0a-33411f8f74e3"). InnerVolumeSpecName "kube-api-access-gkffw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:04.966021 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:04.965978 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-istio-token" pod "ec361aac-99e4-4f3c-9c0a-33411f8f74e3" (UID: "ec361aac-99e4-4f3c-9c0a-33411f8f74e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:05.065079 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:05.065033 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:28:05.065079 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:05.065061 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:28:05.065079 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:05.065070 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:28:05.065079 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:05.065081 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkffw\" (UniqueName: \"kubernetes.io/projected/ec361aac-99e4-4f3c-9c0a-33411f8f74e3-kube-api-access-gkffw\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:28:05.071809 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:05.071780 6221 generic.go:356] "Generic (PLEG): container finished" podID="ec361aac-99e4-4f3c-9c0a-33411f8f74e3" containerID="e2e40b5f725217a45cd267db9bba839c6b2665ca2fa0b45028aabd990910324b" exitCode=0 Oct 30 01:28:05.071949 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:05.071843 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" Oct 30 01:28:05.071949 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:05.071847 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" event={"ID":"ec361aac-99e4-4f3c-9c0a-33411f8f74e3","Type":"ContainerDied","Data":"e2e40b5f725217a45cd267db9bba839c6b2665ca2fa0b45028aabd990910324b"} Oct 30 01:28:05.072028 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:05.071972 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p" event={"ID":"ec361aac-99e4-4f3c-9c0a-33411f8f74e3","Type":"ContainerDied","Data":"3111975d96296cc52f86a33b26521b7824a3485bf80ad7b7f21434fd3452829b"} Oct 30 01:28:05.072028 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:05.072001 6221 scope.go:122] "RemoveContainer" containerID="e2e40b5f725217a45cd267db9bba839c6b2665ca2fa0b45028aabd990910324b" Oct 30 01:28:05.073578 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:05.073553 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-558327-7945456798-frsjf" event={"ID":"4cbfa236-b599-4625-bf0f-8d5d3e47398e","Type":"ContainerStarted","Data":"53d4099edfaf4c0c5620b2809ce001c409af1079d00bef4c1d07d335af9a5668"} Oct 30 01:28:05.073691 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:05.073604 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-558327-7945456798-frsjf" event={"ID":"4cbfa236-b599-4625-bf0f-8d5d3e47398e","Type":"ContainerStarted","Data":"6bd7e065505353fd5a24bd32e76279791c80bbf1cfb51f39ce4a7ba177e8835b"} Oct 30 01:28:05.073691 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:05.073636 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-558327-7945456798-frsjf" Oct 30 01:28:05.081178 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:05.081162 6221 scope.go:122] "RemoveContainer" containerID="24ede9319bbff330d24a8f9e1c538c3dbfd69f8bf0c6c448f05c44837902ef50" Oct 30 01:28:05.088748 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:05.088732 6221 scope.go:122] "RemoveContainer" containerID="e2e40b5f725217a45cd267db9bba839c6b2665ca2fa0b45028aabd990910324b" Oct 30 01:28:05.088992 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:28:05.088975 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2e40b5f725217a45cd267db9bba839c6b2665ca2fa0b45028aabd990910324b\": container with ID starting with e2e40b5f725217a45cd267db9bba839c6b2665ca2fa0b45028aabd990910324b not found: ID does not exist" containerID="e2e40b5f725217a45cd267db9bba839c6b2665ca2fa0b45028aabd990910324b" Oct 30 01:28:05.089039 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:05.088999 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2e40b5f725217a45cd267db9bba839c6b2665ca2fa0b45028aabd990910324b"} err="failed to get container status \"e2e40b5f725217a45cd267db9bba839c6b2665ca2fa0b45028aabd990910324b\": rpc error: code = NotFound desc = could not find container \"e2e40b5f725217a45cd267db9bba839c6b2665ca2fa0b45028aabd990910324b\": container with ID starting with e2e40b5f725217a45cd267db9bba839c6b2665ca2fa0b45028aabd990910324b not found: ID does not exist" Oct 30 01:28:05.089039 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:05.089014 6221 scope.go:122] "RemoveContainer" containerID="24ede9319bbff330d24a8f9e1c538c3dbfd69f8bf0c6c448f05c44837902ef50" Oct 30 01:28:05.089201 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:28:05.089187 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24ede9319bbff330d24a8f9e1c538c3dbfd69f8bf0c6c448f05c44837902ef50\": container with ID starting with 24ede9319bbff330d24a8f9e1c538c3dbfd69f8bf0c6c448f05c44837902ef50 not found: ID does not exist" containerID="24ede9319bbff330d24a8f9e1c538c3dbfd69f8bf0c6c448f05c44837902ef50" Oct 30 01:28:05.089239 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:05.089205 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24ede9319bbff330d24a8f9e1c538c3dbfd69f8bf0c6c448f05c44837902ef50"} err="failed to get container status \"24ede9319bbff330d24a8f9e1c538c3dbfd69f8bf0c6c448f05c44837902ef50\": rpc error: code = NotFound desc = could not find container \"24ede9319bbff330d24a8f9e1c538c3dbfd69f8bf0c6c448f05c44837902ef50\": container with ID starting with 24ede9319bbff330d24a8f9e1c538c3dbfd69f8bf0c6c448f05c44837902ef50 not found: ID does not exist" Oct 30 01:28:05.131597 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:05.131561 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p"] Oct 30 01:28:05.149695 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:05.149161 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-5b724cd8-openshift-default-84b9db95ff-dxp5p"] Oct 30 01:28:05.193173 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:05.193120 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-558327-7945456798-frsjf" podStartSLOduration=1.87522171 podStartE2EDuration="2.193105107s" totalImagesPullingTime="317.883397ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:05.191463131 +0000 UTC m=+608.769750469" watchObservedRunningTime="2026-10-30 01:28:05.193105107 +0000 UTC m=+608.771392443" Oct 30 01:28:06.545368 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:06.545336 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec361aac-99e4-4f3c-9c0a-33411f8f74e3" path="/var/lib/kubelet/pods/ec361aac-99e4-4f3c-9c0a-33411f8f74e3/volumes" Oct 30 01:28:07.531725 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:07.531695 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj"] Oct 30 01:28:07.531934 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:07.531909 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" podUID="03ccf6ef-c34a-4847-9791-1050db4d14d5" containerName="istio-proxy" containerID="cri-o://b92a681a2f3ca74e300f3cd73efe4201ad8a396a01097a2fdad2af7f7e86c1ec" gracePeriod=30 Oct 30 01:28:07.564419 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:07.564390 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-18c05df8-7b97d9bf6f-q95mm"] Oct 30 01:28:07.564861 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:07.564654 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-18c05df8-7b97d9bf6f-q95mm" podUID="cb7fd5cb-4d81-450e-bb74-ae198f698559" containerName="maas-api" containerID="cri-o://a958a37f507527a9b113110e465377c36bed619439ef7ae89aeb77c88e5af261" gracePeriod=30 Oct 30 01:28:07.754643 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:07.754609 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:28:07.816495 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:07.816459 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-18c05df8-7b97d9bf6f-q95mm" Oct 30 01:28:07.899049 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:07.899013 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb7fd5cb-4d81-450e-bb74-ae198f698559-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb7fd5cb-4d81-450e-bb74-ae198f698559-maas-api-tls\") pod \"cb7fd5cb-4d81-450e-bb74-ae198f698559\" (UID: \"cb7fd5cb-4d81-450e-bb74-ae198f698559\") " Oct 30 01:28:07.899198 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:07.899118 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb7fd5cb-4d81-450e-bb74-ae198f698559-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb7fd5cb-4d81-450e-bb74-ae198f698559-metrics-certs\") pod \"cb7fd5cb-4d81-450e-bb74-ae198f698559\" (UID: \"cb7fd5cb-4d81-450e-bb74-ae198f698559\") " Oct 30 01:28:07.899262 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:07.899193 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb7fd5cb-4d81-450e-bb74-ae198f698559-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb7fd5cb-4d81-450e-bb74-ae198f698559-openshift-service-ca\") pod \"cb7fd5cb-4d81-450e-bb74-ae198f698559\" (UID: \"cb7fd5cb-4d81-450e-bb74-ae198f698559\") " Oct 30 01:28:07.899316 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:07.899257 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb7fd5cb-4d81-450e-bb74-ae198f698559-kube-api-access-6tfrn\" (UniqueName: \"kubernetes.io/projected/cb7fd5cb-4d81-450e-bb74-ae198f698559-kube-api-access-6tfrn\") pod \"cb7fd5cb-4d81-450e-bb74-ae198f698559\" (UID: \"cb7fd5cb-4d81-450e-bb74-ae198f698559\") " Oct 30 01:28:07.899596 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:07.899567 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb7fd5cb-4d81-450e-bb74-ae198f698559-openshift-service-ca" pod "cb7fd5cb-4d81-450e-bb74-ae198f698559" (UID: "cb7fd5cb-4d81-450e-bb74-ae198f698559"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:28:07.899742 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:07.899652 6221 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb7fd5cb-4d81-450e-bb74-ae198f698559-openshift-service-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:28:07.901201 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:07.901172 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb7fd5cb-4d81-450e-bb74-ae198f698559-kube-api-access-6tfrn" pod "cb7fd5cb-4d81-450e-bb74-ae198f698559" (UID: "cb7fd5cb-4d81-450e-bb74-ae198f698559"). InnerVolumeSpecName "kube-api-access-6tfrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:07.901201 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:07.901195 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb7fd5cb-4d81-450e-bb74-ae198f698559-metrics-certs" pod "cb7fd5cb-4d81-450e-bb74-ae198f698559" (UID: "cb7fd5cb-4d81-450e-bb74-ae198f698559"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:28:07.901338 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:07.901211 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb7fd5cb-4d81-450e-bb74-ae198f698559-maas-api-tls" pod "cb7fd5cb-4d81-450e-bb74-ae198f698559" (UID: "cb7fd5cb-4d81-450e-bb74-ae198f698559"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:28:08.001056 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:08.001014 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tfrn\" (UniqueName: \"kubernetes.io/projected/cb7fd5cb-4d81-450e-bb74-ae198f698559-kube-api-access-6tfrn\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:28:08.001056 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:08.001059 6221 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb7fd5cb-4d81-450e-bb74-ae198f698559-maas-api-tls\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:28:08.001278 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:08.001074 6221 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb7fd5cb-4d81-450e-bb74-ae198f698559-metrics-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:28:08.091668 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:08.091582 6221 generic.go:356] "Generic (PLEG): container finished" podID="cb7fd5cb-4d81-450e-bb74-ae198f698559" containerID="a958a37f507527a9b113110e465377c36bed619439ef7ae89aeb77c88e5af261" exitCode=0 Oct 30 01:28:08.091668 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:08.091628 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-18c05df8-7b97d9bf6f-q95mm" Oct 30 01:28:08.091850 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:08.091666 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-18c05df8-7b97d9bf6f-q95mm" event={"ID":"cb7fd5cb-4d81-450e-bb74-ae198f698559","Type":"ContainerDied","Data":"a958a37f507527a9b113110e465377c36bed619439ef7ae89aeb77c88e5af261"} Oct 30 01:28:08.091850 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:08.091723 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-18c05df8-7b97d9bf6f-q95mm" event={"ID":"cb7fd5cb-4d81-450e-bb74-ae198f698559","Type":"ContainerDied","Data":"ae00f5b0238c5907525bc7a9d5958526c0bb8c8facbbf38bfd803b86519e966f"} Oct 30 01:28:08.091850 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:08.091747 6221 scope.go:122] "RemoveContainer" containerID="a958a37f507527a9b113110e465377c36bed619439ef7ae89aeb77c88e5af261" Oct 30 01:28:08.101247 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:08.101224 6221 scope.go:122] "RemoveContainer" containerID="a958a37f507527a9b113110e465377c36bed619439ef7ae89aeb77c88e5af261" Oct 30 01:28:08.101545 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:28:08.101513 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a958a37f507527a9b113110e465377c36bed619439ef7ae89aeb77c88e5af261\": container with ID starting with a958a37f507527a9b113110e465377c36bed619439ef7ae89aeb77c88e5af261 not found: ID does not exist" containerID="a958a37f507527a9b113110e465377c36bed619439ef7ae89aeb77c88e5af261" Oct 30 01:28:08.101624 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:08.101555 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a958a37f507527a9b113110e465377c36bed619439ef7ae89aeb77c88e5af261"} err="failed to get container status \"a958a37f507527a9b113110e465377c36bed619439ef7ae89aeb77c88e5af261\": rpc error: code = NotFound desc = could not find container \"a958a37f507527a9b113110e465377c36bed619439ef7ae89aeb77c88e5af261\": container with ID starting with a958a37f507527a9b113110e465377c36bed619439ef7ae89aeb77c88e5af261 not found: ID does not exist" Oct 30 01:28:08.148454 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:08.148430 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-18c05df8-7b97d9bf6f-q95mm"] Oct 30 01:28:08.152157 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:08.152133 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-18c05df8-7b97d9bf6f-q95mm"] Oct 30 01:28:08.267569 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:08.267517 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-eda881-74bfc54cbd-4tc64"] Oct 30 01:28:08.267936 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:08.267923 6221 cpu_manager.go:414] "containerMap: removing container" podUID="ec361aac-99e4-4f3c-9c0a-33411f8f74e3" containerName="istio-proxy" Oct 30 01:28:08.267980 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:08.267937 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec361aac-99e4-4f3c-9c0a-33411f8f74e3" containerName="istio-proxy" Oct 30 01:28:08.267980 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:08.267949 6221 cpu_manager.go:414] "containerMap: removing container" podUID="cb7fd5cb-4d81-450e-bb74-ae198f698559" containerName="maas-api" Oct 30 01:28:08.267980 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:08.267954 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb7fd5cb-4d81-450e-bb74-ae198f698559" containerName="maas-api" Oct 30 01:28:08.267980 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:08.267971 6221 cpu_manager.go:414] "containerMap: removing container" podUID="ec361aac-99e4-4f3c-9c0a-33411f8f74e3" containerName="istio-proxy" Oct 30 01:28:08.267980 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:08.267977 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec361aac-99e4-4f3c-9c0a-33411f8f74e3" containerName="istio-proxy" Oct 30 01:28:08.268130 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:08.268032 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec361aac-99e4-4f3c-9c0a-33411f8f74e3" containerName="istio-proxy" Oct 30 01:28:08.268130 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:08.268043 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec361aac-99e4-4f3c-9c0a-33411f8f74e3" containerName="istio-proxy" Oct 30 01:28:08.268130 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:08.268050 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb7fd5cb-4d81-450e-bb74-ae198f698559" containerName="maas-api" Oct 30 01:28:08.272484 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:08.272463 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eda881-74bfc54cbd-4tc64" Oct 30 01:28:08.293068 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:08.292178 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-eda881-74bfc54cbd-4tc64"] Oct 30 01:28:08.293068 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:08.292776 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-eda881-serving-cert\"" Oct 30 01:28:08.303797 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:08.303768 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c26c4344-4ccb-44d9-ad1d-cfd8720bc56d-metrics-certs\") pod \"maas-api-e2e-worker-w2-eda881-74bfc54cbd-4tc64\" (UID: \"c26c4344-4ccb-44d9-ad1d-cfd8720bc56d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eda881-74bfc54cbd-4tc64" Oct 30 01:28:08.303898 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:08.303841 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98qmf\" (UniqueName: \"kubernetes.io/projected/c26c4344-4ccb-44d9-ad1d-cfd8720bc56d-kube-api-access-98qmf\") pod \"maas-api-e2e-worker-w2-eda881-74bfc54cbd-4tc64\" (UID: \"c26c4344-4ccb-44d9-ad1d-cfd8720bc56d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eda881-74bfc54cbd-4tc64" Oct 30 01:28:08.303954 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:08.303928 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c26c4344-4ccb-44d9-ad1d-cfd8720bc56d-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-eda881-74bfc54cbd-4tc64\" (UID: \"c26c4344-4ccb-44d9-ad1d-cfd8720bc56d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eda881-74bfc54cbd-4tc64" Oct 30 01:28:08.304066 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:08.304044 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c26c4344-4ccb-44d9-ad1d-cfd8720bc56d-maas-api-tls\") pod \"maas-api-e2e-worker-w2-eda881-74bfc54cbd-4tc64\" (UID: \"c26c4344-4ccb-44d9-ad1d-cfd8720bc56d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eda881-74bfc54cbd-4tc64" Oct 30 01:28:08.405307 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:08.405220 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c26c4344-4ccb-44d9-ad1d-cfd8720bc56d-metrics-certs\") pod \"maas-api-e2e-worker-w2-eda881-74bfc54cbd-4tc64\" (UID: \"c26c4344-4ccb-44d9-ad1d-cfd8720bc56d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eda881-74bfc54cbd-4tc64" Oct 30 01:28:08.405307 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:08.405267 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98qmf\" (UniqueName: \"kubernetes.io/projected/c26c4344-4ccb-44d9-ad1d-cfd8720bc56d-kube-api-access-98qmf\") pod \"maas-api-e2e-worker-w2-eda881-74bfc54cbd-4tc64\" (UID: \"c26c4344-4ccb-44d9-ad1d-cfd8720bc56d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eda881-74bfc54cbd-4tc64" Oct 30 01:28:08.405307 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:08.405285 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c26c4344-4ccb-44d9-ad1d-cfd8720bc56d-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-eda881-74bfc54cbd-4tc64\" (UID: \"c26c4344-4ccb-44d9-ad1d-cfd8720bc56d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eda881-74bfc54cbd-4tc64" Oct 30 01:28:08.405672 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:08.405321 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c26c4344-4ccb-44d9-ad1d-cfd8720bc56d-maas-api-tls\") pod \"maas-api-e2e-worker-w2-eda881-74bfc54cbd-4tc64\" (UID: \"c26c4344-4ccb-44d9-ad1d-cfd8720bc56d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eda881-74bfc54cbd-4tc64" Oct 30 01:28:08.405672 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:28:08.405413 6221 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-eda881-serving-cert: secret "maas-api-e2e-worker-w2-eda881-serving-cert" not found Oct 30 01:28:08.405672 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:28:08.405466 6221 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c26c4344-4ccb-44d9-ad1d-cfd8720bc56d-maas-api-tls podName:c26c4344-4ccb-44d9-ad1d-cfd8720bc56d nodeName:}" failed. No retries permitted until 2026-10-30 01:28:08.905449533 +0000 UTC m=+612.483736849 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c26c4344-4ccb-44d9-ad1d-cfd8720bc56d-maas-api-tls") pod "maas-api-e2e-worker-w2-eda881-74bfc54cbd-4tc64" (UID: "c26c4344-4ccb-44d9-ad1d-cfd8720bc56d") : secret "maas-api-e2e-worker-w2-eda881-serving-cert" not found Oct 30 01:28:08.406035 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:08.406002 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c26c4344-4ccb-44d9-ad1d-cfd8720bc56d-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-eda881-74bfc54cbd-4tc64\" (UID: \"c26c4344-4ccb-44d9-ad1d-cfd8720bc56d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eda881-74bfc54cbd-4tc64" Oct 30 01:28:08.407805 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:08.407778 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c26c4344-4ccb-44d9-ad1d-cfd8720bc56d-metrics-certs\") pod \"maas-api-e2e-worker-w2-eda881-74bfc54cbd-4tc64\" (UID: \"c26c4344-4ccb-44d9-ad1d-cfd8720bc56d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eda881-74bfc54cbd-4tc64" Oct 30 01:28:08.426647 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:08.426623 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98qmf\" (UniqueName: \"kubernetes.io/projected/c26c4344-4ccb-44d9-ad1d-cfd8720bc56d-kube-api-access-98qmf\") pod \"maas-api-e2e-worker-w2-eda881-74bfc54cbd-4tc64\" (UID: \"c26c4344-4ccb-44d9-ad1d-cfd8720bc56d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eda881-74bfc54cbd-4tc64" Oct 30 01:28:08.566555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:08.565079 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb7fd5cb-4d81-450e-bb74-ae198f698559" path="/var/lib/kubelet/pods/cb7fd5cb-4d81-450e-bb74-ae198f698559/volumes" Oct 30 01:28:08.922110 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:08.915560 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c26c4344-4ccb-44d9-ad1d-cfd8720bc56d-maas-api-tls\") pod \"maas-api-e2e-worker-w2-eda881-74bfc54cbd-4tc64\" (UID: \"c26c4344-4ccb-44d9-ad1d-cfd8720bc56d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eda881-74bfc54cbd-4tc64" Oct 30 01:28:08.938056 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:08.937208 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c26c4344-4ccb-44d9-ad1d-cfd8720bc56d-maas-api-tls\") pod \"maas-api-e2e-worker-w2-eda881-74bfc54cbd-4tc64\" (UID: \"c26c4344-4ccb-44d9-ad1d-cfd8720bc56d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eda881-74bfc54cbd-4tc64" Oct 30 01:28:09.076358 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.076321 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a3a2ac-6677b6f44c-bpm5m"] Oct 30 01:28:09.081299 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.081275 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a3a2ac-6677b6f44c-bpm5m" Oct 30 01:28:09.098964 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.098931 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a3a2ac-6677b6f44c-bpm5m"] Oct 30 01:28:09.099759 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.099735 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-a3a2ac\"" Oct 30 01:28:09.130554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.122163 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26457d5d-6b8f-4e85-8bc1-321115791efc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-a3a2ac-6677b6f44c-bpm5m\" (UID: \"26457d5d-6b8f-4e85-8bc1-321115791efc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a3a2ac-6677b6f44c-bpm5m" Oct 30 01:28:09.130554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.122241 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8v7c\" (UniqueName: \"kubernetes.io/projected/26457d5d-6b8f-4e85-8bc1-321115791efc-kube-api-access-r8v7c\") pod \"payload-pre-processing-e2e-worker-w0-a3a2ac-6677b6f44c-bpm5m\" (UID: \"26457d5d-6b8f-4e85-8bc1-321115791efc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a3a2ac-6677b6f44c-bpm5m" Oct 30 01:28:09.130554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.122713 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-a3a2ac-dockercfg-9xmbr\"" Oct 30 01:28:09.177524 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.177430 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a3a2ac-5d6ff74d96-dnpzn"] Oct 30 01:28:09.181125 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.181107 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-a3a2ac-5d6ff74d96-dnpzn" Oct 30 01:28:09.185029 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.185005 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eda881-74bfc54cbd-4tc64" Oct 30 01:28:09.200174 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.200148 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a3a2ac-5d6ff74d96-dnpzn"] Oct 30 01:28:09.232555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.232115 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26457d5d-6b8f-4e85-8bc1-321115791efc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-a3a2ac-6677b6f44c-bpm5m\" (UID: \"26457d5d-6b8f-4e85-8bc1-321115791efc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a3a2ac-6677b6f44c-bpm5m" Oct 30 01:28:09.232555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.232190 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8v7c\" (UniqueName: \"kubernetes.io/projected/26457d5d-6b8f-4e85-8bc1-321115791efc-kube-api-access-r8v7c\") pod \"payload-pre-processing-e2e-worker-w0-a3a2ac-6677b6f44c-bpm5m\" (UID: \"26457d5d-6b8f-4e85-8bc1-321115791efc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a3a2ac-6677b6f44c-bpm5m" Oct 30 01:28:09.236552 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.233795 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26457d5d-6b8f-4e85-8bc1-321115791efc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-a3a2ac-6677b6f44c-bpm5m\" (UID: \"26457d5d-6b8f-4e85-8bc1-321115791efc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a3a2ac-6677b6f44c-bpm5m" Oct 30 01:28:09.275715 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.266372 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8v7c\" (UniqueName: \"kubernetes.io/projected/26457d5d-6b8f-4e85-8bc1-321115791efc-kube-api-access-r8v7c\") pod \"payload-pre-processing-e2e-worker-w0-a3a2ac-6677b6f44c-bpm5m\" (UID: \"26457d5d-6b8f-4e85-8bc1-321115791efc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a3a2ac-6677b6f44c-bpm5m" Oct 30 01:28:09.345593 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.344084 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/702e03b2-6e41-4a27-837a-ede7f1c2e2c2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-a3a2ac-5d6ff74d96-dnpzn\" (UID: \"702e03b2-6e41-4a27-837a-ede7f1c2e2c2\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a3a2ac-5d6ff74d96-dnpzn" Oct 30 01:28:09.345593 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.344140 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlqdr\" (UniqueName: \"kubernetes.io/projected/702e03b2-6e41-4a27-837a-ede7f1c2e2c2-kube-api-access-tlqdr\") pod \"payload-processing-e2e-worker-w0-a3a2ac-5d6ff74d96-dnpzn\" (UID: \"702e03b2-6e41-4a27-837a-ede7f1c2e2c2\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a3a2ac-5d6ff74d96-dnpzn" Oct 30 01:28:09.394619 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.394584 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a3a2ac-6677b6f44c-bpm5m" Oct 30 01:28:09.444806 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.444724 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlqdr\" (UniqueName: \"kubernetes.io/projected/702e03b2-6e41-4a27-837a-ede7f1c2e2c2-kube-api-access-tlqdr\") pod \"payload-processing-e2e-worker-w0-a3a2ac-5d6ff74d96-dnpzn\" (UID: \"702e03b2-6e41-4a27-837a-ede7f1c2e2c2\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a3a2ac-5d6ff74d96-dnpzn" Oct 30 01:28:09.444956 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.444846 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/702e03b2-6e41-4a27-837a-ede7f1c2e2c2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-a3a2ac-5d6ff74d96-dnpzn\" (UID: \"702e03b2-6e41-4a27-837a-ede7f1c2e2c2\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a3a2ac-5d6ff74d96-dnpzn" Oct 30 01:28:09.445549 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.445510 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/702e03b2-6e41-4a27-837a-ede7f1c2e2c2-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-a3a2ac-5d6ff74d96-dnpzn\" (UID: \"702e03b2-6e41-4a27-837a-ede7f1c2e2c2\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a3a2ac-5d6ff74d96-dnpzn" Oct 30 01:28:09.465518 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.465486 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlqdr\" (UniqueName: \"kubernetes.io/projected/702e03b2-6e41-4a27-837a-ede7f1c2e2c2-kube-api-access-tlqdr\") pod \"payload-processing-e2e-worker-w0-a3a2ac-5d6ff74d96-dnpzn\" (UID: \"702e03b2-6e41-4a27-837a-ede7f1c2e2c2\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-a3a2ac-5d6ff74d96-dnpzn" Oct 30 01:28:09.493846 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.493817 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-a3a2ac-5d6ff74d96-dnpzn" Oct 30 01:28:09.545712 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.545466 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-eda881-74bfc54cbd-4tc64"] Oct 30 01:28:09.549467 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:28:09.549425 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc26c4344_4ccb_44d9_ad1d_cfd8720bc56d.slice/crio-c89b1f604699e9ad7be29e3eae1c1b69aa9bd687b0ee76a3dc0e4b4adb31d720 WatchSource:0}: Error finding container c89b1f604699e9ad7be29e3eae1c1b69aa9bd687b0ee76a3dc0e4b4adb31d720: Status 404 returned error can't find the container with id c89b1f604699e9ad7be29e3eae1c1b69aa9bd687b0ee76a3dc0e4b4adb31d720 Oct 30 01:28:09.723622 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.723226 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a3a2ac-6677b6f44c-bpm5m"] Oct 30 01:28:09.755007 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.754980 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz"] Oct 30 01:28:09.760644 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.760619 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" Oct 30 01:28:09.777111 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.777083 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-cf82f8-openshift-default-dockercfg-xt5zt\"" Oct 30 01:28:09.794027 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.793997 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz"] Oct 30 01:28:09.858194 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.858162 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32567804-08a2-48d7-942d-bd82afff5728-istiod-ca-cert\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz\" (UID: \"32567804-08a2-48d7-942d-bd82afff5728\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" Oct 30 01:28:09.858194 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.858204 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32567804-08a2-48d7-942d-bd82afff5728-credential-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz\" (UID: \"32567804-08a2-48d7-942d-bd82afff5728\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" Oct 30 01:28:09.858442 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.858229 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32567804-08a2-48d7-942d-bd82afff5728-istio-data\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz\" (UID: \"32567804-08a2-48d7-942d-bd82afff5728\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" Oct 30 01:28:09.858442 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.858344 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32567804-08a2-48d7-942d-bd82afff5728-workload-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz\" (UID: \"32567804-08a2-48d7-942d-bd82afff5728\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" Oct 30 01:28:09.858442 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.858365 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32567804-08a2-48d7-942d-bd82afff5728-istio-token\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz\" (UID: \"32567804-08a2-48d7-942d-bd82afff5728\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" Oct 30 01:28:09.858587 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.858460 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnwkj\" (UniqueName: \"kubernetes.io/projected/32567804-08a2-48d7-942d-bd82afff5728-kube-api-access-vnwkj\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz\" (UID: \"32567804-08a2-48d7-942d-bd82afff5728\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" Oct 30 01:28:09.858587 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.858492 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32567804-08a2-48d7-942d-bd82afff5728-istio-podinfo\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz\" (UID: \"32567804-08a2-48d7-942d-bd82afff5728\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" Oct 30 01:28:09.858587 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.858518 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32567804-08a2-48d7-942d-bd82afff5728-istio-envoy\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz\" (UID: \"32567804-08a2-48d7-942d-bd82afff5728\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" Oct 30 01:28:09.858587 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.858567 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32567804-08a2-48d7-942d-bd82afff5728-workload-certs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz\" (UID: \"32567804-08a2-48d7-942d-bd82afff5728\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" Oct 30 01:28:09.931554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.931169 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a3a2ac-5d6ff74d96-dnpzn"] Oct 30 01:28:09.938024 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:28:09.937979 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod702e03b2_6e41_4a27_837a_ede7f1c2e2c2.slice/crio-e6ab79fc42e4b4458ff57383d381647a6d7e1c4420c53889fc75f8b4edd89be8 WatchSource:0}: Error finding container e6ab79fc42e4b4458ff57383d381647a6d7e1c4420c53889fc75f8b4edd89be8: Status 404 returned error can't find the container with id e6ab79fc42e4b4458ff57383d381647a6d7e1c4420c53889fc75f8b4edd89be8 Oct 30 01:28:09.966692 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.962729 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnwkj\" (UniqueName: \"kubernetes.io/projected/32567804-08a2-48d7-942d-bd82afff5728-kube-api-access-vnwkj\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz\" (UID: \"32567804-08a2-48d7-942d-bd82afff5728\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" Oct 30 01:28:09.966692 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.962781 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32567804-08a2-48d7-942d-bd82afff5728-istio-podinfo\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz\" (UID: \"32567804-08a2-48d7-942d-bd82afff5728\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" Oct 30 01:28:09.966692 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.962819 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32567804-08a2-48d7-942d-bd82afff5728-istio-envoy\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz\" (UID: \"32567804-08a2-48d7-942d-bd82afff5728\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" Oct 30 01:28:09.966692 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.962875 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32567804-08a2-48d7-942d-bd82afff5728-workload-certs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz\" (UID: \"32567804-08a2-48d7-942d-bd82afff5728\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" Oct 30 01:28:09.966692 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.962954 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32567804-08a2-48d7-942d-bd82afff5728-istiod-ca-cert\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz\" (UID: \"32567804-08a2-48d7-942d-bd82afff5728\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" Oct 30 01:28:09.966692 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.963000 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32567804-08a2-48d7-942d-bd82afff5728-credential-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz\" (UID: \"32567804-08a2-48d7-942d-bd82afff5728\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" Oct 30 01:28:09.966692 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.963039 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32567804-08a2-48d7-942d-bd82afff5728-istio-data\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz\" (UID: \"32567804-08a2-48d7-942d-bd82afff5728\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" Oct 30 01:28:09.966692 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.963076 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32567804-08a2-48d7-942d-bd82afff5728-workload-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz\" (UID: \"32567804-08a2-48d7-942d-bd82afff5728\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" Oct 30 01:28:09.966692 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.963115 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32567804-08a2-48d7-942d-bd82afff5728-istio-token\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz\" (UID: \"32567804-08a2-48d7-942d-bd82afff5728\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" Oct 30 01:28:09.966692 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.965349 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32567804-08a2-48d7-942d-bd82afff5728-istiod-ca-cert\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz\" (UID: \"32567804-08a2-48d7-942d-bd82afff5728\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" Oct 30 01:28:09.966692 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.965849 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32567804-08a2-48d7-942d-bd82afff5728-istio-data\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz\" (UID: \"32567804-08a2-48d7-942d-bd82afff5728\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" Oct 30 01:28:09.966692 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.966123 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32567804-08a2-48d7-942d-bd82afff5728-credential-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz\" (UID: \"32567804-08a2-48d7-942d-bd82afff5728\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" Oct 30 01:28:09.966692 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.966378 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32567804-08a2-48d7-942d-bd82afff5728-workload-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz\" (UID: \"32567804-08a2-48d7-942d-bd82afff5728\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" Oct 30 01:28:09.966692 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.966632 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32567804-08a2-48d7-942d-bd82afff5728-workload-certs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz\" (UID: \"32567804-08a2-48d7-942d-bd82afff5728\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" Oct 30 01:28:09.967817 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.967794 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32567804-08a2-48d7-942d-bd82afff5728-istio-podinfo\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz\" (UID: \"32567804-08a2-48d7-942d-bd82afff5728\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" Oct 30 01:28:09.969025 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.968980 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32567804-08a2-48d7-942d-bd82afff5728-istio-envoy\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz\" (UID: \"32567804-08a2-48d7-942d-bd82afff5728\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" Oct 30 01:28:09.989906 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.989824 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32567804-08a2-48d7-942d-bd82afff5728-istio-token\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz\" (UID: \"32567804-08a2-48d7-942d-bd82afff5728\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" Oct 30 01:28:09.990084 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:09.990061 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnwkj\" (UniqueName: \"kubernetes.io/projected/32567804-08a2-48d7-942d-bd82afff5728-kube-api-access-vnwkj\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz\" (UID: \"32567804-08a2-48d7-942d-bd82afff5728\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" Oct 30 01:28:10.098090 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:10.098069 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" Oct 30 01:28:10.131213 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:10.131178 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a3a2ac-6677b6f44c-bpm5m" event={"ID":"26457d5d-6b8f-4e85-8bc1-321115791efc","Type":"ContainerStarted","Data":"cf1461e454b5a0ca17822203ff258a06f43a988b751fd9de7cfae531a4ff22af"} Oct 30 01:28:10.131450 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:10.131423 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a3a2ac-6677b6f44c-bpm5m" event={"ID":"26457d5d-6b8f-4e85-8bc1-321115791efc","Type":"ContainerStarted","Data":"9941e92383ab3d0f214ee1dced98987fbcdd7a3ea916393b98be9649b5addb58"} Oct 30 01:28:10.133559 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:10.132627 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a3a2ac-6677b6f44c-bpm5m" Oct 30 01:28:10.135153 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:10.135087 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eda881-74bfc54cbd-4tc64" event={"ID":"c26c4344-4ccb-44d9-ad1d-cfd8720bc56d","Type":"ContainerStarted","Data":"c89b1f604699e9ad7be29e3eae1c1b69aa9bd687b0ee76a3dc0e4b4adb31d720"} Oct 30 01:28:10.137114 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:10.137093 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-a3a2ac-5d6ff74d96-dnpzn" event={"ID":"702e03b2-6e41-4a27-837a-ede7f1c2e2c2","Type":"ContainerStarted","Data":"dc649fe4c29d4979c29690108ae04f89428ace55d60d815535b1f72a8e0df35f"} Oct 30 01:28:10.137281 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:10.137262 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-a3a2ac-5d6ff74d96-dnpzn" event={"ID":"702e03b2-6e41-4a27-837a-ede7f1c2e2c2","Type":"ContainerStarted","Data":"e6ab79fc42e4b4458ff57383d381647a6d7e1c4420c53889fc75f8b4edd89be8"} Oct 30 01:28:10.137511 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:10.137471 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-a3a2ac-5d6ff74d96-dnpzn" Oct 30 01:28:10.173936 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:10.173875 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a3a2ac-6677b6f44c-bpm5m" podStartSLOduration=1.173857152 podStartE2EDuration="1.173857152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:10.172819692 +0000 UTC m=+613.751107040" watchObservedRunningTime="2026-10-30 01:28:10.173857152 +0000 UTC m=+613.752144489" Oct 30 01:28:10.233599 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:10.231405 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-a3a2ac-5d6ff74d96-dnpzn" podStartSLOduration=1.231384724 podStartE2EDuration="1.231384724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:10.229137006 +0000 UTC m=+613.807424342" watchObservedRunningTime="2026-10-30 01:28:10.231384724 +0000 UTC m=+613.809672063" Oct 30 01:28:10.462016 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:10.461983 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz"] Oct 30 01:28:10.466364 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:28:10.466330 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32567804_08a2_48d7_942d_bd82afff5728.slice/crio-f0cc7ae573acc454605d213778a39320000b312542838189c616d37aacc46df0 WatchSource:0}: Error finding container f0cc7ae573acc454605d213778a39320000b312542838189c616d37aacc46df0: Status 404 returned error can't find the container with id f0cc7ae573acc454605d213778a39320000b312542838189c616d37aacc46df0 Oct 30 01:28:10.468730 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:10.468698 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:28:10.468813 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:10.468759 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:28:10.468813 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:10.468790 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:28:10.780842 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:10.780753 6221 patch_prober.go:29] interesting pod/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:28:10.780842 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:10.780809 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" podUID="03ccf6ef-c34a-4847-9791-1050db4d14d5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:28:11.085352 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:11.085323 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-558327-7945456798-frsjf" Oct 30 01:28:11.146653 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:11.146042 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eda881-74bfc54cbd-4tc64" event={"ID":"c26c4344-4ccb-44d9-ad1d-cfd8720bc56d","Type":"ContainerStarted","Data":"76e28a44d9968b949b498442b4929f86a7d9634b0b960b7fcc938f2a92876385"} Oct 30 01:28:11.146653 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:11.146176 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eda881-74bfc54cbd-4tc64" Oct 30 01:28:11.149435 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:11.148930 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" event={"ID":"32567804-08a2-48d7-942d-bd82afff5728","Type":"ContainerStarted","Data":"6c8adf670d2a303004f1d2686d816378502841ffc9b15f96b962fd2356da0d73"} Oct 30 01:28:11.149435 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:11.148966 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" event={"ID":"32567804-08a2-48d7-942d-bd82afff5728","Type":"ContainerStarted","Data":"f0cc7ae573acc454605d213778a39320000b312542838189c616d37aacc46df0"} Oct 30 01:28:11.191837 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:11.191770 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eda881-74bfc54cbd-4tc64" podStartSLOduration=2.69276652 podStartE2EDuration="3.191756707s" totalImagesPullingTime="498.990187ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:11.189796017 +0000 UTC m=+614.768083354" watchObservedRunningTime="2026-10-30 01:28:11.191756707 +0000 UTC m=+614.770044043" Oct 30 01:28:11.235292 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:11.235227 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" podStartSLOduration=2.2352121990000002 podStartE2EDuration="2.235212199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:11.233067293 +0000 UTC m=+614.811354626" watchObservedRunningTime="2026-10-30 01:28:11.235212199 +0000 UTC m=+614.813499534" Oct 30 01:28:12.070973 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.070941 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a3a2ac-8456757f5d-8xzjx"] Oct 30 01:28:12.074339 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.074319 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a3a2ac-8456757f5d-8xzjx" Oct 30 01:28:12.087184 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.087163 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24f7ec30-91bf-4af8-a913-f1650e340227-metrics-certs\") pod \"maas-api-e2e-worker-w0-a3a2ac-8456757f5d-8xzjx\" (UID: \"24f7ec30-91bf-4af8-a913-f1650e340227\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a3a2ac-8456757f5d-8xzjx" Oct 30 01:28:12.087259 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.087206 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24f7ec30-91bf-4af8-a913-f1650e340227-maas-api-tls\") pod \"maas-api-e2e-worker-w0-a3a2ac-8456757f5d-8xzjx\" (UID: \"24f7ec30-91bf-4af8-a913-f1650e340227\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a3a2ac-8456757f5d-8xzjx" Oct 30 01:28:12.087294 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.087281 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24f7ec30-91bf-4af8-a913-f1650e340227-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-a3a2ac-8456757f5d-8xzjx\" (UID: \"24f7ec30-91bf-4af8-a913-f1650e340227\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a3a2ac-8456757f5d-8xzjx" Oct 30 01:28:12.087329 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.087319 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg6d8\" (UniqueName: \"kubernetes.io/projected/24f7ec30-91bf-4af8-a913-f1650e340227-kube-api-access-cg6d8\") pod \"maas-api-e2e-worker-w0-a3a2ac-8456757f5d-8xzjx\" (UID: \"24f7ec30-91bf-4af8-a913-f1650e340227\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a3a2ac-8456757f5d-8xzjx" Oct 30 01:28:12.098548 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.098507 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" Oct 30 01:28:12.104980 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.104958 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" Oct 30 01:28:12.105758 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.105738 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-a3a2ac-serving-cert\"" Oct 30 01:28:12.132557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.132225 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a3a2ac-8456757f5d-8xzjx"] Oct 30 01:28:12.154678 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.154643 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" Oct 30 01:28:12.155574 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.155556 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" Oct 30 01:28:12.187969 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.187935 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24f7ec30-91bf-4af8-a913-f1650e340227-metrics-certs\") pod \"maas-api-e2e-worker-w0-a3a2ac-8456757f5d-8xzjx\" (UID: \"24f7ec30-91bf-4af8-a913-f1650e340227\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a3a2ac-8456757f5d-8xzjx" Oct 30 01:28:12.188089 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.188003 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24f7ec30-91bf-4af8-a913-f1650e340227-maas-api-tls\") pod \"maas-api-e2e-worker-w0-a3a2ac-8456757f5d-8xzjx\" (UID: \"24f7ec30-91bf-4af8-a913-f1650e340227\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a3a2ac-8456757f5d-8xzjx" Oct 30 01:28:12.188089 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.188046 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24f7ec30-91bf-4af8-a913-f1650e340227-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-a3a2ac-8456757f5d-8xzjx\" (UID: \"24f7ec30-91bf-4af8-a913-f1650e340227\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a3a2ac-8456757f5d-8xzjx" Oct 30 01:28:12.188179 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.188100 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg6d8\" (UniqueName: \"kubernetes.io/projected/24f7ec30-91bf-4af8-a913-f1650e340227-kube-api-access-cg6d8\") pod \"maas-api-e2e-worker-w0-a3a2ac-8456757f5d-8xzjx\" (UID: \"24f7ec30-91bf-4af8-a913-f1650e340227\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a3a2ac-8456757f5d-8xzjx" Oct 30 01:28:12.188347 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:28:12.188325 6221 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-a3a2ac-serving-cert: secret "maas-api-e2e-worker-w0-a3a2ac-serving-cert" not found Oct 30 01:28:12.188442 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:28:12.188400 6221 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24f7ec30-91bf-4af8-a913-f1650e340227-maas-api-tls podName:24f7ec30-91bf-4af8-a913-f1650e340227 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:12.688379914 +0000 UTC m=+616.266667235 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/24f7ec30-91bf-4af8-a913-f1650e340227-maas-api-tls") pod "maas-api-e2e-worker-w0-a3a2ac-8456757f5d-8xzjx" (UID: "24f7ec30-91bf-4af8-a913-f1650e340227") : secret "maas-api-e2e-worker-w0-a3a2ac-serving-cert" not found Oct 30 01:28:12.188751 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.188732 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24f7ec30-91bf-4af8-a913-f1650e340227-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-a3a2ac-8456757f5d-8xzjx\" (UID: \"24f7ec30-91bf-4af8-a913-f1650e340227\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a3a2ac-8456757f5d-8xzjx" Oct 30 01:28:12.190594 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.190574 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24f7ec30-91bf-4af8-a913-f1650e340227-metrics-certs\") pod \"maas-api-e2e-worker-w0-a3a2ac-8456757f5d-8xzjx\" (UID: \"24f7ec30-91bf-4af8-a913-f1650e340227\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a3a2ac-8456757f5d-8xzjx" Oct 30 01:28:12.233291 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.233226 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg6d8\" (UniqueName: \"kubernetes.io/projected/24f7ec30-91bf-4af8-a913-f1650e340227-kube-api-access-cg6d8\") pod \"maas-api-e2e-worker-w0-a3a2ac-8456757f5d-8xzjx\" (UID: \"24f7ec30-91bf-4af8-a913-f1650e340227\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a3a2ac-8456757f5d-8xzjx" Oct 30 01:28:12.692662 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.692622 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24f7ec30-91bf-4af8-a913-f1650e340227-maas-api-tls\") pod \"maas-api-e2e-worker-w0-a3a2ac-8456757f5d-8xzjx\" (UID: \"24f7ec30-91bf-4af8-a913-f1650e340227\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a3a2ac-8456757f5d-8xzjx" Oct 30 01:28:12.695118 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.695084 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24f7ec30-91bf-4af8-a913-f1650e340227-maas-api-tls\") pod \"maas-api-e2e-worker-w0-a3a2ac-8456757f5d-8xzjx\" (UID: \"24f7ec30-91bf-4af8-a913-f1650e340227\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a3a2ac-8456757f5d-8xzjx" Oct 30 01:28:12.790748 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.790723 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" Oct 30 01:28:12.792982 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.792965 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03ccf6ef-c34a-4847-9791-1050db4d14d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03ccf6ef-c34a-4847-9791-1050db4d14d5-istio-podinfo\") pod \"03ccf6ef-c34a-4847-9791-1050db4d14d5\" (UID: \"03ccf6ef-c34a-4847-9791-1050db4d14d5\") " Oct 30 01:28:12.793032 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.792997 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03ccf6ef-c34a-4847-9791-1050db4d14d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03ccf6ef-c34a-4847-9791-1050db4d14d5-istiod-ca-cert\") pod \"03ccf6ef-c34a-4847-9791-1050db4d14d5\" (UID: \"03ccf6ef-c34a-4847-9791-1050db4d14d5\") " Oct 30 01:28:12.793074 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.793029 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03ccf6ef-c34a-4847-9791-1050db4d14d5-kube-api-access-dcv7g\" (UniqueName: \"kubernetes.io/projected/03ccf6ef-c34a-4847-9791-1050db4d14d5-kube-api-access-dcv7g\") pod \"03ccf6ef-c34a-4847-9791-1050db4d14d5\" (UID: \"03ccf6ef-c34a-4847-9791-1050db4d14d5\") " Oct 30 01:28:12.793109 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.793071 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03ccf6ef-c34a-4847-9791-1050db4d14d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03ccf6ef-c34a-4847-9791-1050db4d14d5-workload-certs\") pod \"03ccf6ef-c34a-4847-9791-1050db4d14d5\" (UID: \"03ccf6ef-c34a-4847-9791-1050db4d14d5\") " Oct 30 01:28:12.793109 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.793095 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03ccf6ef-c34a-4847-9791-1050db4d14d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03ccf6ef-c34a-4847-9791-1050db4d14d5-istio-data\") pod \"03ccf6ef-c34a-4847-9791-1050db4d14d5\" (UID: \"03ccf6ef-c34a-4847-9791-1050db4d14d5\") " Oct 30 01:28:12.793196 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.793129 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03ccf6ef-c34a-4847-9791-1050db4d14d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03ccf6ef-c34a-4847-9791-1050db4d14d5-istio-envoy\") pod \"03ccf6ef-c34a-4847-9791-1050db4d14d5\" (UID: \"03ccf6ef-c34a-4847-9791-1050db4d14d5\") " Oct 30 01:28:12.793196 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.793177 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03ccf6ef-c34a-4847-9791-1050db4d14d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ccf6ef-c34a-4847-9791-1050db4d14d5-workload-socket\") pod \"03ccf6ef-c34a-4847-9791-1050db4d14d5\" (UID: \"03ccf6ef-c34a-4847-9791-1050db4d14d5\") " Oct 30 01:28:12.793295 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.793235 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03ccf6ef-c34a-4847-9791-1050db4d14d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ccf6ef-c34a-4847-9791-1050db4d14d5-credential-socket\") pod \"03ccf6ef-c34a-4847-9791-1050db4d14d5\" (UID: \"03ccf6ef-c34a-4847-9791-1050db4d14d5\") " Oct 30 01:28:12.793295 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.793276 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03ccf6ef-c34a-4847-9791-1050db4d14d5-istio-token\" (UniqueName: \"kubernetes.io/projected/03ccf6ef-c34a-4847-9791-1050db4d14d5-istio-token\") pod \"03ccf6ef-c34a-4847-9791-1050db4d14d5\" (UID: \"03ccf6ef-c34a-4847-9791-1050db4d14d5\") " Oct 30 01:28:12.793412 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.793352 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ccf6ef-c34a-4847-9791-1050db4d14d5-workload-certs" pod "03ccf6ef-c34a-4847-9791-1050db4d14d5" (UID: "03ccf6ef-c34a-4847-9791-1050db4d14d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:12.793464 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.793423 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ccf6ef-c34a-4847-9791-1050db4d14d5-istio-data" pod "03ccf6ef-c34a-4847-9791-1050db4d14d5" (UID: "03ccf6ef-c34a-4847-9791-1050db4d14d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:12.793464 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.793429 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ccf6ef-c34a-4847-9791-1050db4d14d5-workload-socket" pod "03ccf6ef-c34a-4847-9791-1050db4d14d5" (UID: "03ccf6ef-c34a-4847-9791-1050db4d14d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:12.793655 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.793627 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ccf6ef-c34a-4847-9791-1050db4d14d5-credential-socket" pod "03ccf6ef-c34a-4847-9791-1050db4d14d5" (UID: "03ccf6ef-c34a-4847-9791-1050db4d14d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:12.793775 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.793704 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03ccf6ef-c34a-4847-9791-1050db4d14d5-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:28:12.793775 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.793726 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03ccf6ef-c34a-4847-9791-1050db4d14d5-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:28:12.793775 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.793741 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ccf6ef-c34a-4847-9791-1050db4d14d5-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:28:12.793775 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.793755 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03ccf6ef-c34a-4847-9791-1050db4d14d5-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:28:12.794000 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.793799 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03ccf6ef-c34a-4847-9791-1050db4d14d5-istiod-ca-cert" pod "03ccf6ef-c34a-4847-9791-1050db4d14d5" (UID: "03ccf6ef-c34a-4847-9791-1050db4d14d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:28:12.795139 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.795089 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03ccf6ef-c34a-4847-9791-1050db4d14d5-istio-envoy" pod "03ccf6ef-c34a-4847-9791-1050db4d14d5" (UID: "03ccf6ef-c34a-4847-9791-1050db4d14d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:28:12.795237 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.795199 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03ccf6ef-c34a-4847-9791-1050db4d14d5-istio-podinfo" pod "03ccf6ef-c34a-4847-9791-1050db4d14d5" (UID: "03ccf6ef-c34a-4847-9791-1050db4d14d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:28:12.795237 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.795225 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03ccf6ef-c34a-4847-9791-1050db4d14d5-kube-api-access-dcv7g" pod "03ccf6ef-c34a-4847-9791-1050db4d14d5" (UID: "03ccf6ef-c34a-4847-9791-1050db4d14d5"). InnerVolumeSpecName "kube-api-access-dcv7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:12.795331 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.795313 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03ccf6ef-c34a-4847-9791-1050db4d14d5-istio-token" pod "03ccf6ef-c34a-4847-9791-1050db4d14d5" (UID: "03ccf6ef-c34a-4847-9791-1050db4d14d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:28:12.894994 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.894957 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03ccf6ef-c34a-4847-9791-1050db4d14d5-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:28:12.894994 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.894986 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03ccf6ef-c34a-4847-9791-1050db4d14d5-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:28:12.894994 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.894996 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03ccf6ef-c34a-4847-9791-1050db4d14d5-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:28:12.895247 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.895004 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcv7g\" (UniqueName: \"kubernetes.io/projected/03ccf6ef-c34a-4847-9791-1050db4d14d5-kube-api-access-dcv7g\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:28:12.895247 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.895014 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03ccf6ef-c34a-4847-9791-1050db4d14d5-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:28:12.987611 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:12.987577 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a3a2ac-8456757f5d-8xzjx" Oct 30 01:28:13.161884 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:13.161850 6221 generic.go:356] "Generic (PLEG): container finished" podID="03ccf6ef-c34a-4847-9791-1050db4d14d5" containerID="b92a681a2f3ca74e300f3cd73efe4201ad8a396a01097a2fdad2af7f7e86c1ec" exitCode=0 Oct 30 01:28:13.162351 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:13.161908 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" Oct 30 01:28:13.162351 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:13.161934 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" event={"ID":"03ccf6ef-c34a-4847-9791-1050db4d14d5","Type":"ContainerDied","Data":"b92a681a2f3ca74e300f3cd73efe4201ad8a396a01097a2fdad2af7f7e86c1ec"} Oct 30 01:28:13.162351 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:13.161996 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj" event={"ID":"03ccf6ef-c34a-4847-9791-1050db4d14d5","Type":"ContainerDied","Data":"be4f44d419c95bb440d772cb8b27672b9d48c0a0ffe6776992893bf2e5e73ba0"} Oct 30 01:28:13.162351 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:13.162028 6221 scope.go:122] "RemoveContainer" containerID="b92a681a2f3ca74e300f3cd73efe4201ad8a396a01097a2fdad2af7f7e86c1ec" Oct 30 01:28:13.171930 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:13.171853 6221 scope.go:122] "RemoveContainer" containerID="f2ac4d800448acebbc51e1f8e6ab4063ade4400929aeecef50b9a112deaa126c" Oct 30 01:28:13.183416 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:13.183375 6221 scope.go:122] "RemoveContainer" containerID="b92a681a2f3ca74e300f3cd73efe4201ad8a396a01097a2fdad2af7f7e86c1ec" Oct 30 01:28:13.183765 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:28:13.183746 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b92a681a2f3ca74e300f3cd73efe4201ad8a396a01097a2fdad2af7f7e86c1ec\": container with ID starting with b92a681a2f3ca74e300f3cd73efe4201ad8a396a01097a2fdad2af7f7e86c1ec not found: ID does not exist" containerID="b92a681a2f3ca74e300f3cd73efe4201ad8a396a01097a2fdad2af7f7e86c1ec" Oct 30 01:28:13.183857 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:13.183776 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b92a681a2f3ca74e300f3cd73efe4201ad8a396a01097a2fdad2af7f7e86c1ec"} err="failed to get container status \"b92a681a2f3ca74e300f3cd73efe4201ad8a396a01097a2fdad2af7f7e86c1ec\": rpc error: code = NotFound desc = could not find container \"b92a681a2f3ca74e300f3cd73efe4201ad8a396a01097a2fdad2af7f7e86c1ec\": container with ID starting with b92a681a2f3ca74e300f3cd73efe4201ad8a396a01097a2fdad2af7f7e86c1ec not found: ID does not exist" Oct 30 01:28:13.183857 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:13.183799 6221 scope.go:122] "RemoveContainer" containerID="f2ac4d800448acebbc51e1f8e6ab4063ade4400929aeecef50b9a112deaa126c" Oct 30 01:28:13.184034 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:28:13.184018 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2ac4d800448acebbc51e1f8e6ab4063ade4400929aeecef50b9a112deaa126c\": container with ID starting with f2ac4d800448acebbc51e1f8e6ab4063ade4400929aeecef50b9a112deaa126c not found: ID does not exist" containerID="f2ac4d800448acebbc51e1f8e6ab4063ade4400929aeecef50b9a112deaa126c" Oct 30 01:28:13.184095 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:13.184041 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2ac4d800448acebbc51e1f8e6ab4063ade4400929aeecef50b9a112deaa126c"} err="failed to get container status \"f2ac4d800448acebbc51e1f8e6ab4063ade4400929aeecef50b9a112deaa126c\": rpc error: code = NotFound desc = could not find container \"f2ac4d800448acebbc51e1f8e6ab4063ade4400929aeecef50b9a112deaa126c\": container with ID starting with f2ac4d800448acebbc51e1f8e6ab4063ade4400929aeecef50b9a112deaa126c not found: ID does not exist" Oct 30 01:28:13.258351 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:13.239088 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj"] Oct 30 01:28:13.264646 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:13.264598 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f29772b9-openshift-default-7949dfcc8f-n9mpj"] Oct 30 01:28:13.333483 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:13.333398 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a3a2ac-8456757f5d-8xzjx"] Oct 30 01:28:13.337682 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:28:13.337657 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24f7ec30_91bf_4af8_a913_f1650e340227.slice/crio-36294fff06d5566ae38b038b69aa3a072c0eafa69c008ae6841cf003ec6cf1a3 WatchSource:0}: Error finding container 36294fff06d5566ae38b038b69aa3a072c0eafa69c008ae6841cf003ec6cf1a3: Status 404 returned error can't find the container with id 36294fff06d5566ae38b038b69aa3a072c0eafa69c008ae6841cf003ec6cf1a3 Oct 30 01:28:14.168352 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:14.168312 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a3a2ac-8456757f5d-8xzjx" event={"ID":"24f7ec30-91bf-4af8-a913-f1650e340227","Type":"ContainerStarted","Data":"848d0f4e8d712e821cbfb0152c347aa883bff13b8400939a4da334a8219e2c62"} Oct 30 01:28:14.168843 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:14.168416 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a3a2ac-8456757f5d-8xzjx" event={"ID":"24f7ec30-91bf-4af8-a913-f1650e340227","Type":"ContainerStarted","Data":"36294fff06d5566ae38b038b69aa3a072c0eafa69c008ae6841cf003ec6cf1a3"} Oct 30 01:28:14.169067 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:14.169039 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a3a2ac-8456757f5d-8xzjx" Oct 30 01:28:14.214964 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:14.214912 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a3a2ac-8456757f5d-8xzjx" podStartSLOduration=1.754265462 podStartE2EDuration="2.214898615s" totalImagesPullingTime="460.633153ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:14.214119658 +0000 UTC m=+617.792406994" watchObservedRunningTime="2026-10-30 01:28:14.214898615 +0000 UTC m=+617.793185951" Oct 30 01:28:14.546519 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:14.546488 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03ccf6ef-c34a-4847-9791-1050db4d14d5" path="/var/lib/kubelet/pods/03ccf6ef-c34a-4847-9791-1050db4d14d5/volumes" Oct 30 01:28:15.078647 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:15.078617 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-558327-bd6c69d5b-scg4b" Oct 30 01:28:16.625496 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:16.625450 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:28:17.167324 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:17.167292 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eda881-74bfc54cbd-4tc64" Oct 30 01:28:18.215962 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:18.215915 6221 generic.go:356] "Generic (PLEG): container finished" podID="666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa" containerID="fc38a75f406fb4f803c813cf9685c74f1f5b1e2808a01db945239b2d6de1db4f" exitCode=0 Oct 30 01:28:18.216440 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:18.215980 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" event={"ID":"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa","Type":"ContainerDied","Data":"fc38a75f406fb4f803c813cf9685c74f1f5b1e2808a01db945239b2d6de1db4f"} Oct 30 01:28:18.216440 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:18.216365 6221 scope.go:122] "RemoveContainer" containerID="fc38a75f406fb4f803c813cf9685c74f1f5b1e2808a01db945239b2d6de1db4f" Oct 30 01:28:18.217685 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:18.217656 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:28:18.217773 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:18.217720 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:28:18.217773 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:18.217751 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:28:19.233920 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:19.233885 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" event={"ID":"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa","Type":"ContainerStarted","Data":"dc349ecb606a610d09839aee538c745dadcc54be91965eb1bd2c6531ca741995"} Oct 30 01:28:19.594367 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:19.594329 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-803472-866c95bf4c-jfvc2"] Oct 30 01:28:19.594813 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:19.594798 6221 cpu_manager.go:414] "containerMap: removing container" podUID="03ccf6ef-c34a-4847-9791-1050db4d14d5" containerName="istio-proxy" Oct 30 01:28:19.594864 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:19.594814 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03ccf6ef-c34a-4847-9791-1050db4d14d5" containerName="istio-proxy" Oct 30 01:28:19.594899 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:19.594893 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="03ccf6ef-c34a-4847-9791-1050db4d14d5" containerName="istio-proxy" Oct 30 01:28:19.594937 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:19.594902 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="03ccf6ef-c34a-4847-9791-1050db4d14d5" containerName="istio-proxy" Oct 30 01:28:19.598219 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:19.598203 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-803472-866c95bf4c-jfvc2" Oct 30 01:28:19.615557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:19.614778 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-803472-serving-cert\"" Oct 30 01:28:19.619557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:19.617671 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-803472-866c95bf4c-jfvc2"] Oct 30 01:28:19.638632 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:19.638596 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7562db53-8060-4df5-a62d-699d3a44caa9-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-803472-866c95bf4c-jfvc2\" (UID: \"7562db53-8060-4df5-a62d-699d3a44caa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-803472-866c95bf4c-jfvc2" Oct 30 01:28:19.638806 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:19.638699 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7562db53-8060-4df5-a62d-699d3a44caa9-metrics-certs\") pod \"maas-api-e2e-worker-w4-803472-866c95bf4c-jfvc2\" (UID: \"7562db53-8060-4df5-a62d-699d3a44caa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-803472-866c95bf4c-jfvc2" Oct 30 01:28:19.638896 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:19.638880 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29zpc\" (UniqueName: \"kubernetes.io/projected/7562db53-8060-4df5-a62d-699d3a44caa9-kube-api-access-29zpc\") pod \"maas-api-e2e-worker-w4-803472-866c95bf4c-jfvc2\" (UID: \"7562db53-8060-4df5-a62d-699d3a44caa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-803472-866c95bf4c-jfvc2" Oct 30 01:28:19.638966 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:19.638952 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7562db53-8060-4df5-a62d-699d3a44caa9-maas-api-tls\") pod \"maas-api-e2e-worker-w4-803472-866c95bf4c-jfvc2\" (UID: \"7562db53-8060-4df5-a62d-699d3a44caa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-803472-866c95bf4c-jfvc2" Oct 30 01:28:19.729021 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:19.728987 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" Oct 30 01:28:19.729408 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:19.729308 6221 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-zwshz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 01:28:19.729668 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:19.729642 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" podUID="666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 01:28:19.739881 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:19.739856 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7562db53-8060-4df5-a62d-699d3a44caa9-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-803472-866c95bf4c-jfvc2\" (UID: \"7562db53-8060-4df5-a62d-699d3a44caa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-803472-866c95bf4c-jfvc2" Oct 30 01:28:19.739982 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:19.739887 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7562db53-8060-4df5-a62d-699d3a44caa9-metrics-certs\") pod \"maas-api-e2e-worker-w4-803472-866c95bf4c-jfvc2\" (UID: \"7562db53-8060-4df5-a62d-699d3a44caa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-803472-866c95bf4c-jfvc2" Oct 30 01:28:19.739982 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:19.739942 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29zpc\" (UniqueName: \"kubernetes.io/projected/7562db53-8060-4df5-a62d-699d3a44caa9-kube-api-access-29zpc\") pod \"maas-api-e2e-worker-w4-803472-866c95bf4c-jfvc2\" (UID: \"7562db53-8060-4df5-a62d-699d3a44caa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-803472-866c95bf4c-jfvc2" Oct 30 01:28:19.739982 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:19.739969 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7562db53-8060-4df5-a62d-699d3a44caa9-maas-api-tls\") pod \"maas-api-e2e-worker-w4-803472-866c95bf4c-jfvc2\" (UID: \"7562db53-8060-4df5-a62d-699d3a44caa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-803472-866c95bf4c-jfvc2" Oct 30 01:28:19.740149 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:28:19.740076 6221 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-803472-serving-cert: secret "maas-api-e2e-worker-w4-803472-serving-cert" not found Oct 30 01:28:19.740149 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:28:19.740131 6221 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7562db53-8060-4df5-a62d-699d3a44caa9-maas-api-tls podName:7562db53-8060-4df5-a62d-699d3a44caa9 nodeName:}" failed. No retries permitted until 2026-10-30 01:28:20.240113608 +0000 UTC m=+623.818400938 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7562db53-8060-4df5-a62d-699d3a44caa9-maas-api-tls") pod "maas-api-e2e-worker-w4-803472-866c95bf4c-jfvc2" (UID: "7562db53-8060-4df5-a62d-699d3a44caa9") : secret "maas-api-e2e-worker-w4-803472-serving-cert" not found Oct 30 01:28:19.740555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:19.740514 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7562db53-8060-4df5-a62d-699d3a44caa9-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-803472-866c95bf4c-jfvc2\" (UID: \"7562db53-8060-4df5-a62d-699d3a44caa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-803472-866c95bf4c-jfvc2" Oct 30 01:28:19.742273 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:19.742254 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7562db53-8060-4df5-a62d-699d3a44caa9-metrics-certs\") pod \"maas-api-e2e-worker-w4-803472-866c95bf4c-jfvc2\" (UID: \"7562db53-8060-4df5-a62d-699d3a44caa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-803472-866c95bf4c-jfvc2" Oct 30 01:28:19.760063 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:19.760041 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29zpc\" (UniqueName: \"kubernetes.io/projected/7562db53-8060-4df5-a62d-699d3a44caa9-kube-api-access-29zpc\") pod \"maas-api-e2e-worker-w4-803472-866c95bf4c-jfvc2\" (UID: \"7562db53-8060-4df5-a62d-699d3a44caa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-803472-866c95bf4c-jfvc2" Oct 30 01:28:20.181000 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:20.180968 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a3a2ac-8456757f5d-8xzjx" Oct 30 01:28:20.245978 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:20.245747 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7562db53-8060-4df5-a62d-699d3a44caa9-maas-api-tls\") pod \"maas-api-e2e-worker-w4-803472-866c95bf4c-jfvc2\" (UID: \"7562db53-8060-4df5-a62d-699d3a44caa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-803472-866c95bf4c-jfvc2" Oct 30 01:28:20.249753 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:20.248071 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7562db53-8060-4df5-a62d-699d3a44caa9-maas-api-tls\") pod \"maas-api-e2e-worker-w4-803472-866c95bf4c-jfvc2\" (UID: \"7562db53-8060-4df5-a62d-699d3a44caa9\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-803472-866c95bf4c-jfvc2" Oct 30 01:28:20.526605 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:20.521702 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-803472-866c95bf4c-jfvc2" Oct 30 01:28:20.720562 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:20.712952 6221 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-zwshz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 01:28:20.720562 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:20.713024 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" podUID="666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 01:28:20.832913 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:20.832879 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-803472-866c95bf4c-jfvc2"] Oct 30 01:28:20.834906 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:28:20.834875 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7562db53_8060_4df5_a62d_699d3a44caa9.slice/crio-74fb9d209a20339c844f9383d5957e1d341c6a9607bcceab2b2bd9367089f45b WatchSource:0}: Error finding container 74fb9d209a20339c844f9383d5957e1d341c6a9607bcceab2b2bd9367089f45b: Status 404 returned error can't find the container with id 74fb9d209a20339c844f9383d5957e1d341c6a9607bcceab2b2bd9367089f45b Oct 30 01:28:20.938332 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:20.938291 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:28:21.152108 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:21.152032 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a3a2ac-6677b6f44c-bpm5m" Oct 30 01:28:21.152434 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:21.152417 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-a3a2ac-5d6ff74d96-dnpzn" Oct 30 01:28:21.246264 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:21.246225 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-803472-866c95bf4c-jfvc2" event={"ID":"7562db53-8060-4df5-a62d-699d3a44caa9","Type":"ContainerStarted","Data":"74fb9d209a20339c844f9383d5957e1d341c6a9607bcceab2b2bd9367089f45b"} Oct 30 01:28:21.507841 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:28:21.507719 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:28:21.734087 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:21.719123 6221 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-zwshz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 01:28:21.734087 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:21.719184 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" podUID="666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 01:28:22.252774 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:22.252740 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-803472-866c95bf4c-jfvc2" event={"ID":"7562db53-8060-4df5-a62d-699d3a44caa9","Type":"ContainerStarted","Data":"495f4da85da3f5a99b78c70f6b35baab54766ac332d0cfb31d9008fee19f93f5"} Oct 30 01:28:22.253242 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:22.252795 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-803472-866c95bf4c-jfvc2" Oct 30 01:28:22.294296 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:22.294248 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-803472-866c95bf4c-jfvc2" podStartSLOduration=2.9709415139999997 podStartE2EDuration="3.294234169s" totalImagesPullingTime="323.292655ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:28:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:28:22.293584385 +0000 UTC m=+625.871871721" watchObservedRunningTime="2026-10-30 01:28:22.294234169 +0000 UTC m=+625.872521504" Oct 30 01:28:22.715351 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:22.715311 6221 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-zwshz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 01:28:22.715673 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:22.715644 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" podUID="666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 01:28:23.718968 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:23.718927 6221 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-zwshz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 01:28:23.719640 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:23.719607 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" podUID="666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 01:28:24.953312 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:24.953279 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" Oct 30 01:28:24.953894 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:24.953867 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" Oct 30 01:28:24.954975 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:24.954951 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" Oct 30 01:28:28.262984 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:28.262955 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-803472-866c95bf4c-jfvc2" Oct 30 01:28:29.658578 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:29.658526 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:28:39.360804 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:39.360722 6221 generic.go:356] "Generic (PLEG): container finished" podID="32567804-08a2-48d7-942d-bd82afff5728" containerID="6c8adf670d2a303004f1d2686d816378502841ffc9b15f96b962fd2356da0d73" exitCode=0 Oct 30 01:28:39.360804 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:39.360777 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" event={"ID":"32567804-08a2-48d7-942d-bd82afff5728","Type":"ContainerDied","Data":"6c8adf670d2a303004f1d2686d816378502841ffc9b15f96b962fd2356da0d73"} Oct 30 01:28:39.361352 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:39.361215 6221 scope.go:122] "RemoveContainer" containerID="6c8adf670d2a303004f1d2686d816378502841ffc9b15f96b962fd2356da0d73" Oct 30 01:28:39.362656 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:39.362628 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:28:39.362734 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:39.362694 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:28:39.362734 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:39.362727 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:28:40.394605 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:40.394563 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" event={"ID":"32567804-08a2-48d7-942d-bd82afff5728","Type":"ContainerStarted","Data":"2c4b408325e40d844e2372d21309d0995f2b4aa0ed497ac2eb02c3f4fe6ba759"} Oct 30 01:28:41.098799 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:41.098761 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" Oct 30 01:28:41.099919 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:41.099861 6221 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 01:28:41.100059 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:41.099955 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" podUID="32567804-08a2-48d7-942d-bd82afff5728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 01:28:41.455186 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:41.455102 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:28:42.099427 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:42.099393 6221 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 01:28:42.099633 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:42.099459 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" podUID="32567804-08a2-48d7-942d-bd82afff5728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 01:28:42.154982 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:42.154941 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" Oct 30 01:28:43.098750 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:43.098714 6221 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 01:28:43.099164 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:43.098779 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" podUID="32567804-08a2-48d7-942d-bd82afff5728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 01:28:44.098522 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:44.098489 6221 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 01:28:44.098713 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:44.098567 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" podUID="32567804-08a2-48d7-942d-bd82afff5728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 01:28:45.098505 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:45.098472 6221 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 01:28:45.098915 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:45.098558 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" podUID="32567804-08a2-48d7-942d-bd82afff5728" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 01:28:46.103156 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:46.103128 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" Oct 30 01:28:46.103993 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:46.103975 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" Oct 30 01:28:49.446507 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:28:49.446473 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:28:51.494096 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:28:51.494062 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:29:00.657290 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:29:00.657256 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:29:07.450381 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:29:07.450347 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl"] Oct 30 01:29:07.451604 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:29:07.451576 6221 cpu_manager.go:414] "containerMap: removing container" podUID="03ccf6ef-c34a-4847-9791-1050db4d14d5" containerName="istio-proxy" Oct 30 01:29:07.451604 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:29:07.451601 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03ccf6ef-c34a-4847-9791-1050db4d14d5" containerName="istio-proxy" Oct 30 01:29:07.456812 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:29:07.456791 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" Oct 30 01:29:07.470784 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:29:07.470754 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl"] Oct 30 01:29:07.481047 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:29:07.481016 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a3a2ac\"/\"openshift-service-ca.crt\"" Oct 30 01:29:07.481207 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:29:07.481099 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a3a2ac\"/\"default-dockercfg-zpcmr\"" Oct 30 01:29:07.483609 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:29:07.483587 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a3a2ac\"/\"premium-w0-a3a2ac-kserve-self-signed-certs\"" Oct 30 01:29:07.483706 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:29:07.483652 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a3a2ac\"/\"kube-root-ca.crt\"" Oct 30 01:29:07.535560 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:29:07.535495 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sffj4\" (UniqueName: \"kubernetes.io/projected/018c3fe2-3e67-45ed-a3d7-d70f3fe5b956-kube-api-access-sffj4\") pod \"premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl\" (UID: \"018c3fe2-3e67-45ed-a3d7-d70f3fe5b956\") " pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" Oct 30 01:29:07.535773 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:29:07.535653 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/018c3fe2-3e67-45ed-a3d7-d70f3fe5b956-tmp-dir\") pod \"premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl\" (UID: \"018c3fe2-3e67-45ed-a3d7-d70f3fe5b956\") " pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" Oct 30 01:29:07.535773 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:29:07.535698 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/018c3fe2-3e67-45ed-a3d7-d70f3fe5b956-model-cache\") pod \"premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl\" (UID: \"018c3fe2-3e67-45ed-a3d7-d70f3fe5b956\") " pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" Oct 30 01:29:07.535878 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:29:07.535852 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/018c3fe2-3e67-45ed-a3d7-d70f3fe5b956-dshm\") pod \"premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl\" (UID: \"018c3fe2-3e67-45ed-a3d7-d70f3fe5b956\") " pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" Oct 30 01:29:07.535917 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:29:07.535891 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/018c3fe2-3e67-45ed-a3d7-d70f3fe5b956-home\") pod \"premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl\" (UID: \"018c3fe2-3e67-45ed-a3d7-d70f3fe5b956\") " pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" Oct 30 01:29:07.535965 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:29:07.535921 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/018c3fe2-3e67-45ed-a3d7-d70f3fe5b956-tls-certs\") pod \"premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl\" (UID: \"018c3fe2-3e67-45ed-a3d7-d70f3fe5b956\") " pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" Oct 30 01:29:07.645398 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:29:07.644164 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/018c3fe2-3e67-45ed-a3d7-d70f3fe5b956-dshm\") pod \"premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl\" (UID: \"018c3fe2-3e67-45ed-a3d7-d70f3fe5b956\") " pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" Oct 30 01:29:07.645398 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:29:07.644229 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/018c3fe2-3e67-45ed-a3d7-d70f3fe5b956-home\") pod \"premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl\" (UID: \"018c3fe2-3e67-45ed-a3d7-d70f3fe5b956\") " pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" Oct 30 01:29:07.645398 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:29:07.644280 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/018c3fe2-3e67-45ed-a3d7-d70f3fe5b956-tls-certs\") pod \"premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl\" (UID: \"018c3fe2-3e67-45ed-a3d7-d70f3fe5b956\") " pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" Oct 30 01:29:07.645398 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:29:07.644316 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sffj4\" (UniqueName: \"kubernetes.io/projected/018c3fe2-3e67-45ed-a3d7-d70f3fe5b956-kube-api-access-sffj4\") pod \"premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl\" (UID: \"018c3fe2-3e67-45ed-a3d7-d70f3fe5b956\") " pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" Oct 30 01:29:07.645398 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:29:07.644369 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/018c3fe2-3e67-45ed-a3d7-d70f3fe5b956-tmp-dir\") pod \"premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl\" (UID: \"018c3fe2-3e67-45ed-a3d7-d70f3fe5b956\") " pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" Oct 30 01:29:07.645398 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:29:07.644427 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/018c3fe2-3e67-45ed-a3d7-d70f3fe5b956-model-cache\") pod \"premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl\" (UID: \"018c3fe2-3e67-45ed-a3d7-d70f3fe5b956\") " pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" Oct 30 01:29:07.645398 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:29:07.644976 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/018c3fe2-3e67-45ed-a3d7-d70f3fe5b956-model-cache\") pod \"premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl\" (UID: \"018c3fe2-3e67-45ed-a3d7-d70f3fe5b956\") " pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" Oct 30 01:29:07.648973 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:29:07.648332 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/018c3fe2-3e67-45ed-a3d7-d70f3fe5b956-dshm\") pod \"premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl\" (UID: \"018c3fe2-3e67-45ed-a3d7-d70f3fe5b956\") " pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" Oct 30 01:29:07.648973 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:29:07.648619 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/018c3fe2-3e67-45ed-a3d7-d70f3fe5b956-home\") pod \"premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl\" (UID: \"018c3fe2-3e67-45ed-a3d7-d70f3fe5b956\") " pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" Oct 30 01:29:07.651642 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:29:07.651617 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/018c3fe2-3e67-45ed-a3d7-d70f3fe5b956-tmp-dir\") pod \"premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl\" (UID: \"018c3fe2-3e67-45ed-a3d7-d70f3fe5b956\") " pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" Oct 30 01:29:07.658747 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:29:07.658693 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/018c3fe2-3e67-45ed-a3d7-d70f3fe5b956-tls-certs\") pod \"premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl\" (UID: \"018c3fe2-3e67-45ed-a3d7-d70f3fe5b956\") " pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" Oct 30 01:29:07.685267 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:29:07.685235 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sffj4\" (UniqueName: \"kubernetes.io/projected/018c3fe2-3e67-45ed-a3d7-d70f3fe5b956-kube-api-access-sffj4\") pod \"premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl\" (UID: \"018c3fe2-3e67-45ed-a3d7-d70f3fe5b956\") " pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" Oct 30 01:29:07.771992 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:29:07.771903 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" Oct 30 01:29:08.071266 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:29:08.071228 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl"] Oct 30 01:29:08.075192 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:29:08.075163 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod018c3fe2_3e67_45ed_a3d7_d70f3fe5b956.slice/crio-2f29ede5aef7e955a1072fcff850ee1b354808104ebeb00e53ec3002b69fc2d2 WatchSource:0}: Error finding container 2f29ede5aef7e955a1072fcff850ee1b354808104ebeb00e53ec3002b69fc2d2: Status 404 returned error can't find the container with id 2f29ede5aef7e955a1072fcff850ee1b354808104ebeb00e53ec3002b69fc2d2 Oct 30 01:29:08.568133 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:29:08.568098 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" event={"ID":"018c3fe2-3e67-45ed-a3d7-d70f3fe5b956","Type":"ContainerStarted","Data":"48289d04014d15ff43e0a04e923abf65a2f96480856367ffb54716448d8ac737"} Oct 30 01:29:08.568552 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:29:08.568144 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" event={"ID":"018c3fe2-3e67-45ed-a3d7-d70f3fe5b956","Type":"ContainerStarted","Data":"2f29ede5aef7e955a1072fcff850ee1b354808104ebeb00e53ec3002b69fc2d2"} Oct 30 01:29:08.607178 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:29:08.607112 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" podStartSLOduration=1.607064121 podStartE2EDuration="1.607064121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:29:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:29:08.605726039 +0000 UTC m=+672.184013375" watchObservedRunningTime="2026-10-30 01:29:08.607064121 +0000 UTC m=+672.185351458" Oct 30 01:29:08.772032 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:29:08.772001 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" Oct 30 01:29:12.757421 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:29:12.757383 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:29:17.772595 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:29:17.772552 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" Oct 30 01:29:17.784889 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:29:17.784867 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" Oct 30 01:29:18.625126 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:29:18.625096 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" Oct 30 01:29:21.493396 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:29:21.493344 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:29:28.877740 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:29:28.877697 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:29:48.264050 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:29:48.264012 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:29:51.493353 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:29:51.493318 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:30:13.829894 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:13.829807 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h"] Oct 30 01:30:13.833644 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:13.833621 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h" Oct 30 01:30:13.848664 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:13.848633 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-cf82f8\"/\"openshift-service-ca.crt\"" Oct 30 01:30:13.861067 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:13.861032 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-cf82f8\"/\"kube-root-ca.crt\"" Oct 30 01:30:13.861239 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:13.861064 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-cf82f8\"/\"default-dockercfg-d6pv6\"" Oct 30 01:30:13.861239 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:13.861222 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-cf82f8\"/\"e2e-conflict-a-d3477dc4-kserve-self-signed-certs\"" Oct 30 01:30:13.863960 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:13.863935 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h"] Oct 30 01:30:13.907144 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:13.907109 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e23cbbfb-1ff9-4233-8367-0bd6e3f642dd-home\") pod \"e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h\" (UID: \"e23cbbfb-1ff9-4233-8367-0bd6e3f642dd\") " pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h" Oct 30 01:30:13.907322 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:13.907154 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e23cbbfb-1ff9-4233-8367-0bd6e3f642dd-dshm\") pod \"e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h\" (UID: \"e23cbbfb-1ff9-4233-8367-0bd6e3f642dd\") " pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h" Oct 30 01:30:13.907373 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:13.907323 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e23cbbfb-1ff9-4233-8367-0bd6e3f642dd-tmp-dir\") pod \"e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h\" (UID: \"e23cbbfb-1ff9-4233-8367-0bd6e3f642dd\") " pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h" Oct 30 01:30:13.907373 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:13.907364 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e23cbbfb-1ff9-4233-8367-0bd6e3f642dd-tls-certs\") pod \"e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h\" (UID: \"e23cbbfb-1ff9-4233-8367-0bd6e3f642dd\") " pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h" Oct 30 01:30:13.907461 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:13.907381 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e23cbbfb-1ff9-4233-8367-0bd6e3f642dd-model-cache\") pod \"e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h\" (UID: \"e23cbbfb-1ff9-4233-8367-0bd6e3f642dd\") " pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h" Oct 30 01:30:13.907461 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:13.907445 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d56jv\" (UniqueName: \"kubernetes.io/projected/e23cbbfb-1ff9-4233-8367-0bd6e3f642dd-kube-api-access-d56jv\") pod \"e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h\" (UID: \"e23cbbfb-1ff9-4233-8367-0bd6e3f642dd\") " pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h" Oct 30 01:30:14.008686 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:14.008638 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e23cbbfb-1ff9-4233-8367-0bd6e3f642dd-dshm\") pod \"e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h\" (UID: \"e23cbbfb-1ff9-4233-8367-0bd6e3f642dd\") " pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h" Oct 30 01:30:14.008885 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:14.008772 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e23cbbfb-1ff9-4233-8367-0bd6e3f642dd-tmp-dir\") pod \"e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h\" (UID: \"e23cbbfb-1ff9-4233-8367-0bd6e3f642dd\") " pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h" Oct 30 01:30:14.008885 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:14.008810 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e23cbbfb-1ff9-4233-8367-0bd6e3f642dd-tls-certs\") pod \"e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h\" (UID: \"e23cbbfb-1ff9-4233-8367-0bd6e3f642dd\") " pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h" Oct 30 01:30:14.008885 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:14.008825 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e23cbbfb-1ff9-4233-8367-0bd6e3f642dd-model-cache\") pod \"e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h\" (UID: \"e23cbbfb-1ff9-4233-8367-0bd6e3f642dd\") " pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h" Oct 30 01:30:14.008885 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:14.008848 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d56jv\" (UniqueName: \"kubernetes.io/projected/e23cbbfb-1ff9-4233-8367-0bd6e3f642dd-kube-api-access-d56jv\") pod \"e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h\" (UID: \"e23cbbfb-1ff9-4233-8367-0bd6e3f642dd\") " pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h" Oct 30 01:30:14.008885 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:14.008883 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e23cbbfb-1ff9-4233-8367-0bd6e3f642dd-home\") pod \"e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h\" (UID: \"e23cbbfb-1ff9-4233-8367-0bd6e3f642dd\") " pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h" Oct 30 01:30:14.009323 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:14.009295 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e23cbbfb-1ff9-4233-8367-0bd6e3f642dd-tmp-dir\") pod \"e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h\" (UID: \"e23cbbfb-1ff9-4233-8367-0bd6e3f642dd\") " pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h" Oct 30 01:30:14.009508 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:14.009343 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e23cbbfb-1ff9-4233-8367-0bd6e3f642dd-home\") pod \"e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h\" (UID: \"e23cbbfb-1ff9-4233-8367-0bd6e3f642dd\") " pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h" Oct 30 01:30:14.009640 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:14.009589 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e23cbbfb-1ff9-4233-8367-0bd6e3f642dd-model-cache\") pod \"e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h\" (UID: \"e23cbbfb-1ff9-4233-8367-0bd6e3f642dd\") " pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h" Oct 30 01:30:14.011514 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:14.011491 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e23cbbfb-1ff9-4233-8367-0bd6e3f642dd-dshm\") pod \"e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h\" (UID: \"e23cbbfb-1ff9-4233-8367-0bd6e3f642dd\") " pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h" Oct 30 01:30:14.018589 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:14.017345 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e23cbbfb-1ff9-4233-8367-0bd6e3f642dd-tls-certs\") pod \"e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h\" (UID: \"e23cbbfb-1ff9-4233-8367-0bd6e3f642dd\") " pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h" Oct 30 01:30:14.033955 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:14.033916 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d56jv\" (UniqueName: \"kubernetes.io/projected/e23cbbfb-1ff9-4233-8367-0bd6e3f642dd-kube-api-access-d56jv\") pod \"e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h\" (UID: \"e23cbbfb-1ff9-4233-8367-0bd6e3f642dd\") " pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h" Oct 30 01:30:14.145632 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:14.145516 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h" Oct 30 01:30:14.451097 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:14.450997 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h"] Oct 30 01:30:14.454330 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:30:14.454306 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode23cbbfb_1ff9_4233_8367_0bd6e3f642dd.slice/crio-e47f116ec88776e54fdbf93d36c89589dde6564df78323b9584fe036985fc504 WatchSource:0}: Error finding container e47f116ec88776e54fdbf93d36c89589dde6564df78323b9584fe036985fc504: Status 404 returned error can't find the container with id e47f116ec88776e54fdbf93d36c89589dde6564df78323b9584fe036985fc504 Oct 30 01:30:14.915001 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:14.914957 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h" event={"ID":"e23cbbfb-1ff9-4233-8367-0bd6e3f642dd","Type":"ContainerStarted","Data":"0460029c212eb6712651c7ea5cdc7d09d4605e9a34bbdc42dc82f2b16f6130de"} Oct 30 01:30:14.915470 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:14.915007 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h" event={"ID":"e23cbbfb-1ff9-4233-8367-0bd6e3f642dd","Type":"ContainerStarted","Data":"e47f116ec88776e54fdbf93d36c89589dde6564df78323b9584fe036985fc504"} Oct 30 01:30:14.962707 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:14.962651 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h" podStartSLOduration=1.962635857 podStartE2EDuration="1.962635857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:14.961594782 +0000 UTC m=+738.539882116" watchObservedRunningTime="2026-10-30 01:30:14.962635857 +0000 UTC m=+738.540923192" Oct 30 01:30:15.146471 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:15.146441 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h" Oct 30 01:30:16.418310 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:16.418277 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr"] Oct 30 01:30:16.422851 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:16.422829 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr" Oct 30 01:30:16.438674 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:16.438647 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-eda881-openshift-default-dockercfg-jwbsd\"" Oct 30 01:30:16.453224 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:16.453188 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr"] Oct 30 01:30:16.534630 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:16.534595 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43cff085-9d72-449b-a7f2-e9033b919e1f-istiod-ca-cert\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr\" (UID: \"43cff085-9d72-449b-a7f2-e9033b919e1f\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr" Oct 30 01:30:16.534803 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:16.534635 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cff085-9d72-449b-a7f2-e9033b919e1f-credential-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr\" (UID: \"43cff085-9d72-449b-a7f2-e9033b919e1f\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr" Oct 30 01:30:16.534803 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:16.534705 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cff085-9d72-449b-a7f2-e9033b919e1f-workload-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr\" (UID: \"43cff085-9d72-449b-a7f2-e9033b919e1f\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr" Oct 30 01:30:16.534803 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:16.534727 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43cff085-9d72-449b-a7f2-e9033b919e1f-istio-podinfo\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr\" (UID: \"43cff085-9d72-449b-a7f2-e9033b919e1f\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr" Oct 30 01:30:16.534803 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:16.534753 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43cff085-9d72-449b-a7f2-e9033b919e1f-istio-data\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr\" (UID: \"43cff085-9d72-449b-a7f2-e9033b919e1f\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr" Oct 30 01:30:16.535024 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:16.534897 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43cff085-9d72-449b-a7f2-e9033b919e1f-workload-certs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr\" (UID: \"43cff085-9d72-449b-a7f2-e9033b919e1f\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr" Oct 30 01:30:16.535024 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:16.534998 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43cff085-9d72-449b-a7f2-e9033b919e1f-istio-token\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr\" (UID: \"43cff085-9d72-449b-a7f2-e9033b919e1f\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr" Oct 30 01:30:16.535129 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:16.535054 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk6f5\" (UniqueName: \"kubernetes.io/projected/43cff085-9d72-449b-a7f2-e9033b919e1f-kube-api-access-xk6f5\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr\" (UID: \"43cff085-9d72-449b-a7f2-e9033b919e1f\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr" Oct 30 01:30:16.535183 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:16.535171 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43cff085-9d72-449b-a7f2-e9033b919e1f-istio-envoy\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr\" (UID: \"43cff085-9d72-449b-a7f2-e9033b919e1f\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr" Oct 30 01:30:16.636072 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:16.636038 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43cff085-9d72-449b-a7f2-e9033b919e1f-istio-envoy\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr\" (UID: \"43cff085-9d72-449b-a7f2-e9033b919e1f\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr" Oct 30 01:30:16.636257 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:16.636119 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43cff085-9d72-449b-a7f2-e9033b919e1f-istiod-ca-cert\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr\" (UID: \"43cff085-9d72-449b-a7f2-e9033b919e1f\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr" Oct 30 01:30:16.636257 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:16.636151 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cff085-9d72-449b-a7f2-e9033b919e1f-credential-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr\" (UID: \"43cff085-9d72-449b-a7f2-e9033b919e1f\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr" Oct 30 01:30:16.636257 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:16.636180 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cff085-9d72-449b-a7f2-e9033b919e1f-workload-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr\" (UID: \"43cff085-9d72-449b-a7f2-e9033b919e1f\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr" Oct 30 01:30:16.636257 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:16.636202 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43cff085-9d72-449b-a7f2-e9033b919e1f-istio-podinfo\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr\" (UID: \"43cff085-9d72-449b-a7f2-e9033b919e1f\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr" Oct 30 01:30:16.636257 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:16.636229 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43cff085-9d72-449b-a7f2-e9033b919e1f-istio-data\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr\" (UID: \"43cff085-9d72-449b-a7f2-e9033b919e1f\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr" Oct 30 01:30:16.636573 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:16.636286 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43cff085-9d72-449b-a7f2-e9033b919e1f-workload-certs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr\" (UID: \"43cff085-9d72-449b-a7f2-e9033b919e1f\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr" Oct 30 01:30:16.636573 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:16.636381 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43cff085-9d72-449b-a7f2-e9033b919e1f-istio-token\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr\" (UID: \"43cff085-9d72-449b-a7f2-e9033b919e1f\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr" Oct 30 01:30:16.636573 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:16.636440 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk6f5\" (UniqueName: \"kubernetes.io/projected/43cff085-9d72-449b-a7f2-e9033b919e1f-kube-api-access-xk6f5\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr\" (UID: \"43cff085-9d72-449b-a7f2-e9033b919e1f\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr" Oct 30 01:30:16.636823 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:16.636796 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cff085-9d72-449b-a7f2-e9033b919e1f-workload-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr\" (UID: \"43cff085-9d72-449b-a7f2-e9033b919e1f\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr" Oct 30 01:30:16.637005 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:16.636821 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cff085-9d72-449b-a7f2-e9033b919e1f-credential-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr\" (UID: \"43cff085-9d72-449b-a7f2-e9033b919e1f\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr" Oct 30 01:30:16.637005 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:16.636934 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43cff085-9d72-449b-a7f2-e9033b919e1f-istiod-ca-cert\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr\" (UID: \"43cff085-9d72-449b-a7f2-e9033b919e1f\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr" Oct 30 01:30:16.637153 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:16.637022 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43cff085-9d72-449b-a7f2-e9033b919e1f-workload-certs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr\" (UID: \"43cff085-9d72-449b-a7f2-e9033b919e1f\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr" Oct 30 01:30:16.637153 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:16.637063 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43cff085-9d72-449b-a7f2-e9033b919e1f-istio-data\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr\" (UID: \"43cff085-9d72-449b-a7f2-e9033b919e1f\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr" Oct 30 01:30:16.638355 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:16.638337 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43cff085-9d72-449b-a7f2-e9033b919e1f-istio-envoy\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr\" (UID: \"43cff085-9d72-449b-a7f2-e9033b919e1f\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr" Oct 30 01:30:16.638827 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:16.638806 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43cff085-9d72-449b-a7f2-e9033b919e1f-istio-podinfo\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr\" (UID: \"43cff085-9d72-449b-a7f2-e9033b919e1f\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr" Oct 30 01:30:16.659224 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:16.659192 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43cff085-9d72-449b-a7f2-e9033b919e1f-istio-token\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr\" (UID: \"43cff085-9d72-449b-a7f2-e9033b919e1f\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr" Oct 30 01:30:16.659352 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:16.659333 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk6f5\" (UniqueName: \"kubernetes.io/projected/43cff085-9d72-449b-a7f2-e9033b919e1f-kube-api-access-xk6f5\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr\" (UID: \"43cff085-9d72-449b-a7f2-e9033b919e1f\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr" Oct 30 01:30:16.738491 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:16.738406 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr" Oct 30 01:30:17.062765 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:17.062725 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr"] Oct 30 01:30:17.086964 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:17.086915 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:30:17.087102 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:17.087020 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:30:17.087102 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:17.087062 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:30:17.944279 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:17.944233 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr" event={"ID":"43cff085-9d72-449b-a7f2-e9033b919e1f","Type":"ContainerStarted","Data":"f5d49aa71eb691fec202516948d393a28786182c737ab92b295f016696b957f9"} Oct 30 01:30:17.944727 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:17.944294 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr" event={"ID":"43cff085-9d72-449b-a7f2-e9033b919e1f","Type":"ContainerStarted","Data":"c5b5b8328038d0f6b051b15976ba86fdb31c87adbb35b784055f49ee394f38ef"} Oct 30 01:30:18.005449 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:18.005397 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr" podStartSLOduration=2.005372131 podStartE2EDuration="2.005372131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:18.005085461 +0000 UTC m=+741.583372802" watchObservedRunningTime="2026-10-30 01:30:18.005372131 +0000 UTC m=+741.583659472" Oct 30 01:30:18.739352 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:18.739315 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr" Oct 30 01:30:18.741019 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:18.740993 6221 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 01:30:18.741090 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:18.741044 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr" podUID="43cff085-9d72-449b-a7f2-e9033b919e1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 01:30:19.739152 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:19.739109 6221 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 01:30:19.739696 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:19.739174 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr" podUID="43cff085-9d72-449b-a7f2-e9033b919e1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 01:30:20.739079 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:20.739039 6221 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 01:30:20.739271 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:20.739098 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr" podUID="43cff085-9d72-449b-a7f2-e9033b919e1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 01:30:21.495206 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:30:21.495165 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:30:21.739110 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:21.739079 6221 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 01:30:21.739277 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:21.739145 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr" podUID="43cff085-9d72-449b-a7f2-e9033b919e1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 01:30:22.634503 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:22.634462 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:30:22.739203 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:22.739168 6221 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Oct 30 01:30:22.739377 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:22.739223 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr" podUID="43cff085-9d72-449b-a7f2-e9033b919e1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Oct 30 01:30:24.146733 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:24.146692 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h" Oct 30 01:30:24.165315 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:24.165284 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h" Oct 30 01:30:24.462950 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:24.462867 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr" Oct 30 01:30:24.463223 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:24.463198 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr" Oct 30 01:30:24.463945 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:24.463928 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr" Oct 30 01:30:24.993689 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:24.993659 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h" Oct 30 01:30:40.193387 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:40.193344 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:30:41.688723 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:41.688688 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz"] Oct 30 01:30:41.692450 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:41.692429 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" Oct 30 01:30:41.724048 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:41.724019 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-558327\"/\"default-dockercfg-bz6bf\"" Oct 30 01:30:41.724190 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:41.724088 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-558327\"/\"distinct-w3-558327-kserve-self-signed-certs\"" Oct 30 01:30:41.725861 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:41.725828 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz"] Oct 30 01:30:41.729734 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:41.729713 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-558327\"/\"kube-root-ca.crt\"" Oct 30 01:30:41.729844 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:41.729747 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-558327\"/\"openshift-service-ca.crt\"" Oct 30 01:30:41.744928 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:41.744898 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/354ec282-b972-4fca-b4e3-1e33ef461f2f-tmp-dir\") pod \"distinct-w3-558327-kserve-7df6f6f9d6-5vzrz\" (UID: \"354ec282-b972-4fca-b4e3-1e33ef461f2f\") " pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" Oct 30 01:30:41.745090 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:41.744943 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qgmt\" (UniqueName: \"kubernetes.io/projected/354ec282-b972-4fca-b4e3-1e33ef461f2f-kube-api-access-7qgmt\") pod \"distinct-w3-558327-kserve-7df6f6f9d6-5vzrz\" (UID: \"354ec282-b972-4fca-b4e3-1e33ef461f2f\") " pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" Oct 30 01:30:41.745090 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:41.745011 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/354ec282-b972-4fca-b4e3-1e33ef461f2f-model-cache\") pod \"distinct-w3-558327-kserve-7df6f6f9d6-5vzrz\" (UID: \"354ec282-b972-4fca-b4e3-1e33ef461f2f\") " pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" Oct 30 01:30:41.745229 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:41.745185 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/354ec282-b972-4fca-b4e3-1e33ef461f2f-dshm\") pod \"distinct-w3-558327-kserve-7df6f6f9d6-5vzrz\" (UID: \"354ec282-b972-4fca-b4e3-1e33ef461f2f\") " pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" Oct 30 01:30:41.745229 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:41.745212 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/354ec282-b972-4fca-b4e3-1e33ef461f2f-home\") pod \"distinct-w3-558327-kserve-7df6f6f9d6-5vzrz\" (UID: \"354ec282-b972-4fca-b4e3-1e33ef461f2f\") " pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" Oct 30 01:30:41.745341 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:41.745269 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/354ec282-b972-4fca-b4e3-1e33ef461f2f-tls-certs\") pod \"distinct-w3-558327-kserve-7df6f6f9d6-5vzrz\" (UID: \"354ec282-b972-4fca-b4e3-1e33ef461f2f\") " pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" Oct 30 01:30:41.845902 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:41.845867 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/354ec282-b972-4fca-b4e3-1e33ef461f2f-home\") pod \"distinct-w3-558327-kserve-7df6f6f9d6-5vzrz\" (UID: \"354ec282-b972-4fca-b4e3-1e33ef461f2f\") " pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" Oct 30 01:30:41.846115 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:41.845912 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/354ec282-b972-4fca-b4e3-1e33ef461f2f-tls-certs\") pod \"distinct-w3-558327-kserve-7df6f6f9d6-5vzrz\" (UID: \"354ec282-b972-4fca-b4e3-1e33ef461f2f\") " pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" Oct 30 01:30:41.846115 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:41.845983 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/354ec282-b972-4fca-b4e3-1e33ef461f2f-tmp-dir\") pod \"distinct-w3-558327-kserve-7df6f6f9d6-5vzrz\" (UID: \"354ec282-b972-4fca-b4e3-1e33ef461f2f\") " pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" Oct 30 01:30:41.846115 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:41.846014 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qgmt\" (UniqueName: \"kubernetes.io/projected/354ec282-b972-4fca-b4e3-1e33ef461f2f-kube-api-access-7qgmt\") pod \"distinct-w3-558327-kserve-7df6f6f9d6-5vzrz\" (UID: \"354ec282-b972-4fca-b4e3-1e33ef461f2f\") " pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" Oct 30 01:30:41.846115 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:41.846041 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/354ec282-b972-4fca-b4e3-1e33ef461f2f-model-cache\") pod \"distinct-w3-558327-kserve-7df6f6f9d6-5vzrz\" (UID: \"354ec282-b972-4fca-b4e3-1e33ef461f2f\") " pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" Oct 30 01:30:41.846352 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:41.846142 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/354ec282-b972-4fca-b4e3-1e33ef461f2f-dshm\") pod \"distinct-w3-558327-kserve-7df6f6f9d6-5vzrz\" (UID: \"354ec282-b972-4fca-b4e3-1e33ef461f2f\") " pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" Oct 30 01:30:41.846416 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:41.846369 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/354ec282-b972-4fca-b4e3-1e33ef461f2f-home\") pod \"distinct-w3-558327-kserve-7df6f6f9d6-5vzrz\" (UID: \"354ec282-b972-4fca-b4e3-1e33ef461f2f\") " pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" Oct 30 01:30:41.846416 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:41.846397 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/354ec282-b972-4fca-b4e3-1e33ef461f2f-tmp-dir\") pod \"distinct-w3-558327-kserve-7df6f6f9d6-5vzrz\" (UID: \"354ec282-b972-4fca-b4e3-1e33ef461f2f\") " pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" Oct 30 01:30:41.846489 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:41.846463 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/354ec282-b972-4fca-b4e3-1e33ef461f2f-model-cache\") pod \"distinct-w3-558327-kserve-7df6f6f9d6-5vzrz\" (UID: \"354ec282-b972-4fca-b4e3-1e33ef461f2f\") " pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" Oct 30 01:30:41.848437 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:41.848415 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/354ec282-b972-4fca-b4e3-1e33ef461f2f-dshm\") pod \"distinct-w3-558327-kserve-7df6f6f9d6-5vzrz\" (UID: \"354ec282-b972-4fca-b4e3-1e33ef461f2f\") " pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" Oct 30 01:30:41.848707 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:41.848690 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/354ec282-b972-4fca-b4e3-1e33ef461f2f-tls-certs\") pod \"distinct-w3-558327-kserve-7df6f6f9d6-5vzrz\" (UID: \"354ec282-b972-4fca-b4e3-1e33ef461f2f\") " pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" Oct 30 01:30:41.869520 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:41.869495 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qgmt\" (UniqueName: \"kubernetes.io/projected/354ec282-b972-4fca-b4e3-1e33ef461f2f-kube-api-access-7qgmt\") pod \"distinct-w3-558327-kserve-7df6f6f9d6-5vzrz\" (UID: \"354ec282-b972-4fca-b4e3-1e33ef461f2f\") " pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" Oct 30 01:30:42.005626 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:42.005526 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" Oct 30 01:30:42.326963 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:42.326924 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz"] Oct 30 01:30:42.331077 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:30:42.331047 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod354ec282_b972_4fca_b4e3_1e33ef461f2f.slice/crio-42e36407ed599fa6ade43a38526b945b4b0a525f30dc3dff65a58ec2f1816308 WatchSource:0}: Error finding container 42e36407ed599fa6ade43a38526b945b4b0a525f30dc3dff65a58ec2f1816308: Status 404 returned error can't find the container with id 42e36407ed599fa6ade43a38526b945b4b0a525f30dc3dff65a58ec2f1816308 Oct 30 01:30:43.072587 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:43.072552 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" event={"ID":"354ec282-b972-4fca-b4e3-1e33ef461f2f","Type":"ContainerStarted","Data":"65caa84f2e160303cc7c87a7886dac96fe8738d0a8dd595b9c4c2372d9100601"} Oct 30 01:30:43.072997 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:43.072595 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" event={"ID":"354ec282-b972-4fca-b4e3-1e33ef461f2f","Type":"ContainerStarted","Data":"42e36407ed599fa6ade43a38526b945b4b0a525f30dc3dff65a58ec2f1816308"} Oct 30 01:30:43.124363 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:43.124308 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" podStartSLOduration=2.124295751 podStartE2EDuration="2.124295751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:30:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:30:43.123406592 +0000 UTC m=+766.701693929" watchObservedRunningTime="2026-10-30 01:30:43.124295751 +0000 UTC m=+766.702583065" Oct 30 01:30:44.006225 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:44.006192 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" Oct 30 01:30:51.496205 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:30:51.496168 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:30:52.006005 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:52.005972 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" Oct 30 01:30:52.018862 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:52.018830 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" Oct 30 01:30:52.128017 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:30:52.127978 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" Oct 30 01:31:01.148566 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:01.148510 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:31:01.676734 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:01.676692 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz"] Oct 30 01:31:01.682835 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:01.682806 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz" Oct 30 01:31:01.719913 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:01.719883 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz"] Oct 30 01:31:01.796799 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:01.796756 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jpnm\" (UniqueName: \"kubernetes.io/projected/12605667-e208-4a48-bb1b-61d1e76066bf-kube-api-access-7jpnm\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz\" (UID: \"12605667-e208-4a48-bb1b-61d1e76066bf\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz" Oct 30 01:31:01.796999 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:01.796804 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12605667-e208-4a48-bb1b-61d1e76066bf-workload-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz\" (UID: \"12605667-e208-4a48-bb1b-61d1e76066bf\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz" Oct 30 01:31:01.796999 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:01.796899 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12605667-e208-4a48-bb1b-61d1e76066bf-credential-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz\" (UID: \"12605667-e208-4a48-bb1b-61d1e76066bf\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz" Oct 30 01:31:01.796999 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:01.796985 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12605667-e208-4a48-bb1b-61d1e76066bf-istio-podinfo\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz\" (UID: \"12605667-e208-4a48-bb1b-61d1e76066bf\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz" Oct 30 01:31:01.797130 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:01.797039 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12605667-e208-4a48-bb1b-61d1e76066bf-istio-data\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz\" (UID: \"12605667-e208-4a48-bb1b-61d1e76066bf\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz" Oct 30 01:31:01.797130 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:01.797096 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12605667-e208-4a48-bb1b-61d1e76066bf-istio-envoy\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz\" (UID: \"12605667-e208-4a48-bb1b-61d1e76066bf\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz" Oct 30 01:31:01.797130 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:01.797124 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12605667-e208-4a48-bb1b-61d1e76066bf-istio-token\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz\" (UID: \"12605667-e208-4a48-bb1b-61d1e76066bf\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz" Oct 30 01:31:01.797268 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:01.797226 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12605667-e208-4a48-bb1b-61d1e76066bf-istiod-ca-cert\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz\" (UID: \"12605667-e208-4a48-bb1b-61d1e76066bf\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz" Oct 30 01:31:01.797329 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:01.797311 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12605667-e208-4a48-bb1b-61d1e76066bf-workload-certs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz\" (UID: \"12605667-e208-4a48-bb1b-61d1e76066bf\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz" Oct 30 01:31:01.898417 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:01.898379 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12605667-e208-4a48-bb1b-61d1e76066bf-credential-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz\" (UID: \"12605667-e208-4a48-bb1b-61d1e76066bf\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz" Oct 30 01:31:01.898641 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:01.898433 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12605667-e208-4a48-bb1b-61d1e76066bf-istio-podinfo\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz\" (UID: \"12605667-e208-4a48-bb1b-61d1e76066bf\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz" Oct 30 01:31:01.898641 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:01.898453 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12605667-e208-4a48-bb1b-61d1e76066bf-istio-data\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz\" (UID: \"12605667-e208-4a48-bb1b-61d1e76066bf\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz" Oct 30 01:31:01.898641 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:01.898501 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12605667-e208-4a48-bb1b-61d1e76066bf-istio-envoy\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz\" (UID: \"12605667-e208-4a48-bb1b-61d1e76066bf\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz" Oct 30 01:31:01.898641 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:01.898526 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12605667-e208-4a48-bb1b-61d1e76066bf-istio-token\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz\" (UID: \"12605667-e208-4a48-bb1b-61d1e76066bf\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz" Oct 30 01:31:01.898641 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:01.898621 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12605667-e208-4a48-bb1b-61d1e76066bf-istiod-ca-cert\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz\" (UID: \"12605667-e208-4a48-bb1b-61d1e76066bf\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz" Oct 30 01:31:01.898921 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:01.898670 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12605667-e208-4a48-bb1b-61d1e76066bf-workload-certs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz\" (UID: \"12605667-e208-4a48-bb1b-61d1e76066bf\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz" Oct 30 01:31:01.898921 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:01.898721 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jpnm\" (UniqueName: \"kubernetes.io/projected/12605667-e208-4a48-bb1b-61d1e76066bf-kube-api-access-7jpnm\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz\" (UID: \"12605667-e208-4a48-bb1b-61d1e76066bf\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz" Oct 30 01:31:01.898921 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:01.898751 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12605667-e208-4a48-bb1b-61d1e76066bf-workload-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz\" (UID: \"12605667-e208-4a48-bb1b-61d1e76066bf\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz" Oct 30 01:31:01.899121 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:01.899097 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12605667-e208-4a48-bb1b-61d1e76066bf-workload-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz\" (UID: \"12605667-e208-4a48-bb1b-61d1e76066bf\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz" Oct 30 01:31:01.899186 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:01.899123 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12605667-e208-4a48-bb1b-61d1e76066bf-workload-certs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz\" (UID: \"12605667-e208-4a48-bb1b-61d1e76066bf\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz" Oct 30 01:31:01.899276 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:01.899254 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12605667-e208-4a48-bb1b-61d1e76066bf-istio-data\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz\" (UID: \"12605667-e208-4a48-bb1b-61d1e76066bf\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz" Oct 30 01:31:01.899407 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:01.899389 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12605667-e208-4a48-bb1b-61d1e76066bf-credential-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz\" (UID: \"12605667-e208-4a48-bb1b-61d1e76066bf\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz" Oct 30 01:31:01.899486 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:01.899452 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12605667-e208-4a48-bb1b-61d1e76066bf-istiod-ca-cert\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz\" (UID: \"12605667-e208-4a48-bb1b-61d1e76066bf\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz" Oct 30 01:31:01.901176 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:01.901149 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12605667-e208-4a48-bb1b-61d1e76066bf-istio-envoy\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz\" (UID: \"12605667-e208-4a48-bb1b-61d1e76066bf\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz" Oct 30 01:31:01.901416 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:01.901382 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12605667-e208-4a48-bb1b-61d1e76066bf-istio-podinfo\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz\" (UID: \"12605667-e208-4a48-bb1b-61d1e76066bf\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz" Oct 30 01:31:01.924579 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:01.924522 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12605667-e208-4a48-bb1b-61d1e76066bf-istio-token\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz\" (UID: \"12605667-e208-4a48-bb1b-61d1e76066bf\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz" Oct 30 01:31:01.926496 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:01.926472 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jpnm\" (UniqueName: \"kubernetes.io/projected/12605667-e208-4a48-bb1b-61d1e76066bf-kube-api-access-7jpnm\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz\" (UID: \"12605667-e208-4a48-bb1b-61d1e76066bf\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz" Oct 30 01:31:01.998362 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:01.998278 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz" Oct 30 01:31:02.379127 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:02.376901 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz"] Oct 30 01:31:02.383336 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:02.383299 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:31:02.383602 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:02.383576 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:31:02.383760 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:02.383627 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:31:03.179032 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:03.178991 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz" event={"ID":"12605667-e208-4a48-bb1b-61d1e76066bf","Type":"ContainerStarted","Data":"694a9016590c12c573bc946f60effc9d71f0397f73b59ab6588b47fba1fe6ed2"} Oct 30 01:31:03.179223 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:03.179046 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz" event={"ID":"12605667-e208-4a48-bb1b-61d1e76066bf","Type":"ContainerStarted","Data":"60d64cd87f3ab5f263f1b6e8819ced495fbe40b6338ea7baaeb3ddb744550e9d"} Oct 30 01:31:03.253981 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:03.253926 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz" podStartSLOduration=2.253910889 podStartE2EDuration="2.253910889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:31:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:31:03.240493229 +0000 UTC m=+786.818780566" watchObservedRunningTime="2026-10-30 01:31:03.253910889 +0000 UTC m=+786.832198224" Oct 30 01:31:03.999144 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:03.999102 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz" Oct 30 01:31:04.000738 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:04.000709 6221 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 01:31:04.000865 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:04.000769 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz" podUID="12605667-e208-4a48-bb1b-61d1e76066bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 01:31:04.998712 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:04.998677 6221 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 01:31:04.998888 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:04.998730 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz" podUID="12605667-e208-4a48-bb1b-61d1e76066bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 01:31:05.999381 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:05.999349 6221 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 01:31:05.999811 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:05.999422 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz" podUID="12605667-e208-4a48-bb1b-61d1e76066bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 01:31:06.999607 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:06.999572 6221 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 01:31:07.000037 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:06.999627 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz" podUID="12605667-e208-4a48-bb1b-61d1e76066bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 01:31:07.999460 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:07.999429 6221 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 01:31:07.999654 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:07.999504 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz" podUID="12605667-e208-4a48-bb1b-61d1e76066bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 01:31:08.998771 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:08.998735 6221 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 01:31:08.998955 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:08.998796 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz" podUID="12605667-e208-4a48-bb1b-61d1e76066bf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 01:31:10.003292 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:10.003262 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz" Oct 30 01:31:10.003728 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:10.003554 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz" Oct 30 01:31:10.004368 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:10.004351 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz" Oct 30 01:31:21.494145 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:31:21.494099 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:31:22.955687 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:22.955647 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92"] Oct 30 01:31:22.963430 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:22.963399 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" Oct 30 01:31:22.985844 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:22.985814 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92"] Oct 30 01:31:22.995732 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:22.995702 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-558327\"/\"distinct-2-w3-558327-kserve-self-signed-certs\"" Oct 30 01:31:23.065544 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:23.065502 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea9583aa-8259-4c9b-932e-2d6559008d86-home\") pod \"distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92\" (UID: \"ea9583aa-8259-4c9b-932e-2d6559008d86\") " pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" Oct 30 01:31:23.065734 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:23.065572 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea9583aa-8259-4c9b-932e-2d6559008d86-dshm\") pod \"distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92\" (UID: \"ea9583aa-8259-4c9b-932e-2d6559008d86\") " pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" Oct 30 01:31:23.065734 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:23.065676 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhwmq\" (UniqueName: \"kubernetes.io/projected/ea9583aa-8259-4c9b-932e-2d6559008d86-kube-api-access-lhwmq\") pod \"distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92\" (UID: \"ea9583aa-8259-4c9b-932e-2d6559008d86\") " pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" Oct 30 01:31:23.065734 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:23.065718 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea9583aa-8259-4c9b-932e-2d6559008d86-tls-certs\") pod \"distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92\" (UID: \"ea9583aa-8259-4c9b-932e-2d6559008d86\") " pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" Oct 30 01:31:23.065864 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:23.065798 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea9583aa-8259-4c9b-932e-2d6559008d86-model-cache\") pod \"distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92\" (UID: \"ea9583aa-8259-4c9b-932e-2d6559008d86\") " pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" Oct 30 01:31:23.065864 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:23.065823 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea9583aa-8259-4c9b-932e-2d6559008d86-tmp-dir\") pod \"distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92\" (UID: \"ea9583aa-8259-4c9b-932e-2d6559008d86\") " pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" Oct 30 01:31:23.183553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:23.180874 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea9583aa-8259-4c9b-932e-2d6559008d86-home\") pod \"distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92\" (UID: \"ea9583aa-8259-4c9b-932e-2d6559008d86\") " pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" Oct 30 01:31:23.183553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:23.180938 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea9583aa-8259-4c9b-932e-2d6559008d86-dshm\") pod \"distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92\" (UID: \"ea9583aa-8259-4c9b-932e-2d6559008d86\") " pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" Oct 30 01:31:23.183553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:23.180993 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhwmq\" (UniqueName: \"kubernetes.io/projected/ea9583aa-8259-4c9b-932e-2d6559008d86-kube-api-access-lhwmq\") pod \"distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92\" (UID: \"ea9583aa-8259-4c9b-932e-2d6559008d86\") " pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" Oct 30 01:31:23.183553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:23.181038 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea9583aa-8259-4c9b-932e-2d6559008d86-tls-certs\") pod \"distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92\" (UID: \"ea9583aa-8259-4c9b-932e-2d6559008d86\") " pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" Oct 30 01:31:23.183553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:23.181114 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea9583aa-8259-4c9b-932e-2d6559008d86-model-cache\") pod \"distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92\" (UID: \"ea9583aa-8259-4c9b-932e-2d6559008d86\") " pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" Oct 30 01:31:23.183553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:23.181153 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea9583aa-8259-4c9b-932e-2d6559008d86-tmp-dir\") pod \"distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92\" (UID: \"ea9583aa-8259-4c9b-932e-2d6559008d86\") " pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" Oct 30 01:31:23.183553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:23.181850 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea9583aa-8259-4c9b-932e-2d6559008d86-tmp-dir\") pod \"distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92\" (UID: \"ea9583aa-8259-4c9b-932e-2d6559008d86\") " pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" Oct 30 01:31:23.183553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:23.182144 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea9583aa-8259-4c9b-932e-2d6559008d86-home\") pod \"distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92\" (UID: \"ea9583aa-8259-4c9b-932e-2d6559008d86\") " pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" Oct 30 01:31:23.183553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:23.183266 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea9583aa-8259-4c9b-932e-2d6559008d86-model-cache\") pod \"distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92\" (UID: \"ea9583aa-8259-4c9b-932e-2d6559008d86\") " pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" Oct 30 01:31:23.214676 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:23.211862 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea9583aa-8259-4c9b-932e-2d6559008d86-tls-certs\") pod \"distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92\" (UID: \"ea9583aa-8259-4c9b-932e-2d6559008d86\") " pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" Oct 30 01:31:23.217304 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:23.217272 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea9583aa-8259-4c9b-932e-2d6559008d86-dshm\") pod \"distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92\" (UID: \"ea9583aa-8259-4c9b-932e-2d6559008d86\") " pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" Oct 30 01:31:23.219789 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:23.219598 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhwmq\" (UniqueName: \"kubernetes.io/projected/ea9583aa-8259-4c9b-932e-2d6559008d86-kube-api-access-lhwmq\") pod \"distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92\" (UID: \"ea9583aa-8259-4c9b-932e-2d6559008d86\") " pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" Oct 30 01:31:23.278569 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:23.278516 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" Oct 30 01:31:23.584623 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:23.584588 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92"] Oct 30 01:31:23.587674 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:31:23.587636 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea9583aa_8259_4c9b_932e_2d6559008d86.slice/crio-273265d7aef835e93e9ab136ec3542f2a46b4c15ac6416e358a87cf8c6e9241f WatchSource:0}: Error finding container 273265d7aef835e93e9ab136ec3542f2a46b4c15ac6416e358a87cf8c6e9241f: Status 404 returned error can't find the container with id 273265d7aef835e93e9ab136ec3542f2a46b4c15ac6416e358a87cf8c6e9241f Oct 30 01:31:24.310200 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:24.310166 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" event={"ID":"ea9583aa-8259-4c9b-932e-2d6559008d86","Type":"ContainerStarted","Data":"695ecc7193684eab12d84f19975b89e440ddee0391e3d1dd630856d4ace3125d"} Oct 30 01:31:24.310618 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:24.310210 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" event={"ID":"ea9583aa-8259-4c9b-932e-2d6559008d86","Type":"ContainerStarted","Data":"273265d7aef835e93e9ab136ec3542f2a46b4c15ac6416e358a87cf8c6e9241f"} Oct 30 01:31:24.354952 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:24.354904 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" podStartSLOduration=2.3548911710000002 podStartE2EDuration="2.354891171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:31:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:31:24.352500169 +0000 UTC m=+807.930787506" watchObservedRunningTime="2026-10-30 01:31:24.354891171 +0000 UTC m=+807.933178507" Oct 30 01:31:24.586810 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:24.586724 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q"] Oct 30 01:31:24.590350 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:24.590326 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q" Oct 30 01:31:24.609368 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:24.609345 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-803472-openshift-default-dockercfg-hqdb6\"" Oct 30 01:31:24.628640 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:24.628617 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q"] Oct 30 01:31:24.698796 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:24.698763 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73b85ea4-7469-459c-8ccd-3e9c669bac1c-workload-certs\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q\" (UID: \"73b85ea4-7469-459c-8ccd-3e9c669bac1c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q" Oct 30 01:31:24.698796 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:24.698796 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73b85ea4-7469-459c-8ccd-3e9c669bac1c-istio-envoy\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q\" (UID: \"73b85ea4-7469-459c-8ccd-3e9c669bac1c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q" Oct 30 01:31:24.699009 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:24.698907 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b85ea4-7469-459c-8ccd-3e9c669bac1c-credential-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q\" (UID: \"73b85ea4-7469-459c-8ccd-3e9c669bac1c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q" Oct 30 01:31:24.699049 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:24.699025 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73b85ea4-7469-459c-8ccd-3e9c669bac1c-istio-podinfo\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q\" (UID: \"73b85ea4-7469-459c-8ccd-3e9c669bac1c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q" Oct 30 01:31:24.699102 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:24.699087 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73b85ea4-7469-459c-8ccd-3e9c669bac1c-istio-token\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q\" (UID: \"73b85ea4-7469-459c-8ccd-3e9c669bac1c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q" Oct 30 01:31:24.699138 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:24.699113 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73b85ea4-7469-459c-8ccd-3e9c669bac1c-istiod-ca-cert\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q\" (UID: \"73b85ea4-7469-459c-8ccd-3e9c669bac1c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q" Oct 30 01:31:24.699201 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:24.699190 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73b85ea4-7469-459c-8ccd-3e9c669bac1c-istio-data\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q\" (UID: \"73b85ea4-7469-459c-8ccd-3e9c669bac1c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q" Oct 30 01:31:24.699242 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:24.699226 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b85ea4-7469-459c-8ccd-3e9c669bac1c-workload-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q\" (UID: \"73b85ea4-7469-459c-8ccd-3e9c669bac1c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q" Oct 30 01:31:24.699358 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:24.699338 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pr8bk\" (UniqueName: \"kubernetes.io/projected/73b85ea4-7469-459c-8ccd-3e9c669bac1c-kube-api-access-pr8bk\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q\" (UID: \"73b85ea4-7469-459c-8ccd-3e9c669bac1c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q" Oct 30 01:31:24.800186 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:24.800153 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73b85ea4-7469-459c-8ccd-3e9c669bac1c-istio-token\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q\" (UID: \"73b85ea4-7469-459c-8ccd-3e9c669bac1c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q" Oct 30 01:31:24.800186 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:24.800187 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73b85ea4-7469-459c-8ccd-3e9c669bac1c-istiod-ca-cert\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q\" (UID: \"73b85ea4-7469-459c-8ccd-3e9c669bac1c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q" Oct 30 01:31:24.800430 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:24.800227 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73b85ea4-7469-459c-8ccd-3e9c669bac1c-istio-data\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q\" (UID: \"73b85ea4-7469-459c-8ccd-3e9c669bac1c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q" Oct 30 01:31:24.800430 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:24.800247 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b85ea4-7469-459c-8ccd-3e9c669bac1c-workload-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q\" (UID: \"73b85ea4-7469-459c-8ccd-3e9c669bac1c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q" Oct 30 01:31:24.800430 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:24.800270 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pr8bk\" (UniqueName: \"kubernetes.io/projected/73b85ea4-7469-459c-8ccd-3e9c669bac1c-kube-api-access-pr8bk\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q\" (UID: \"73b85ea4-7469-459c-8ccd-3e9c669bac1c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q" Oct 30 01:31:24.800430 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:24.800319 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73b85ea4-7469-459c-8ccd-3e9c669bac1c-workload-certs\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q\" (UID: \"73b85ea4-7469-459c-8ccd-3e9c669bac1c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q" Oct 30 01:31:24.800430 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:24.800395 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73b85ea4-7469-459c-8ccd-3e9c669bac1c-istio-envoy\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q\" (UID: \"73b85ea4-7469-459c-8ccd-3e9c669bac1c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q" Oct 30 01:31:24.800737 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:24.800445 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b85ea4-7469-459c-8ccd-3e9c669bac1c-credential-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q\" (UID: \"73b85ea4-7469-459c-8ccd-3e9c669bac1c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q" Oct 30 01:31:24.800737 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:24.800511 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73b85ea4-7469-459c-8ccd-3e9c669bac1c-istio-podinfo\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q\" (UID: \"73b85ea4-7469-459c-8ccd-3e9c669bac1c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q" Oct 30 01:31:24.800737 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:24.800668 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73b85ea4-7469-459c-8ccd-3e9c669bac1c-istio-data\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q\" (UID: \"73b85ea4-7469-459c-8ccd-3e9c669bac1c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q" Oct 30 01:31:24.800909 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:24.800788 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b85ea4-7469-459c-8ccd-3e9c669bac1c-workload-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q\" (UID: \"73b85ea4-7469-459c-8ccd-3e9c669bac1c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q" Oct 30 01:31:24.800909 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:24.800844 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73b85ea4-7469-459c-8ccd-3e9c669bac1c-workload-certs\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q\" (UID: \"73b85ea4-7469-459c-8ccd-3e9c669bac1c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q" Oct 30 01:31:24.801026 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:24.800908 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b85ea4-7469-459c-8ccd-3e9c669bac1c-credential-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q\" (UID: \"73b85ea4-7469-459c-8ccd-3e9c669bac1c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q" Oct 30 01:31:24.801115 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:24.801091 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73b85ea4-7469-459c-8ccd-3e9c669bac1c-istiod-ca-cert\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q\" (UID: \"73b85ea4-7469-459c-8ccd-3e9c669bac1c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q" Oct 30 01:31:24.802760 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:24.802734 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73b85ea4-7469-459c-8ccd-3e9c669bac1c-istio-envoy\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q\" (UID: \"73b85ea4-7469-459c-8ccd-3e9c669bac1c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q" Oct 30 01:31:24.802912 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:24.802893 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73b85ea4-7469-459c-8ccd-3e9c669bac1c-istio-podinfo\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q\" (UID: \"73b85ea4-7469-459c-8ccd-3e9c669bac1c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q" Oct 30 01:31:24.819884 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:24.819858 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73b85ea4-7469-459c-8ccd-3e9c669bac1c-istio-token\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q\" (UID: \"73b85ea4-7469-459c-8ccd-3e9c669bac1c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q" Oct 30 01:31:24.819998 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:24.819945 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pr8bk\" (UniqueName: \"kubernetes.io/projected/73b85ea4-7469-459c-8ccd-3e9c669bac1c-kube-api-access-pr8bk\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q\" (UID: \"73b85ea4-7469-459c-8ccd-3e9c669bac1c\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q" Oct 30 01:31:24.903574 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:24.903478 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q" Oct 30 01:31:25.226170 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:25.226146 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q"] Oct 30 01:31:25.228750 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:31:25.228718 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73b85ea4_7469_459c_8ccd_3e9c669bac1c.slice/crio-8c2cff409bc319a537c152ba56b8698f2e0f4cfe126408499295ce0ab7076c7e WatchSource:0}: Error finding container 8c2cff409bc319a537c152ba56b8698f2e0f4cfe126408499295ce0ab7076c7e: Status 404 returned error can't find the container with id 8c2cff409bc319a537c152ba56b8698f2e0f4cfe126408499295ce0ab7076c7e Oct 30 01:31:25.231246 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:25.231210 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:31:25.231322 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:25.231299 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:31:25.231360 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:25.231341 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:31:25.279010 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:25.278990 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" Oct 30 01:31:25.317585 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:25.317555 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q" event={"ID":"73b85ea4-7469-459c-8ccd-3e9c669bac1c","Type":"ContainerStarted","Data":"97db6495da9d4546cbbfc7e92f7595502c258f1d9328e55fe9716867575b14e6"} Oct 30 01:31:25.318008 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:25.317607 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q" event={"ID":"73b85ea4-7469-459c-8ccd-3e9c669bac1c","Type":"ContainerStarted","Data":"8c2cff409bc319a537c152ba56b8698f2e0f4cfe126408499295ce0ab7076c7e"} Oct 30 01:31:25.362505 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:25.362440 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q" podStartSLOduration=1.362412786 podStartE2EDuration="1.362412786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:31:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:31:25.360157769 +0000 UTC m=+808.938445105" watchObservedRunningTime="2026-10-30 01:31:25.362412786 +0000 UTC m=+808.940700123" Oct 30 01:31:25.718559 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:25.717456 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:31:25.904322 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:25.904280 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q" Oct 30 01:31:26.908887 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:26.908851 6221 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 01:31:26.918893 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:26.918849 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q" podUID="73b85ea4-7469-459c-8ccd-3e9c669bac1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 01:31:27.904249 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:27.904210 6221 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 01:31:27.904437 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:27.904271 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q" podUID="73b85ea4-7469-459c-8ccd-3e9c669bac1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 01:31:28.904292 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:28.904254 6221 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 01:31:28.904744 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:28.904316 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q" podUID="73b85ea4-7469-459c-8ccd-3e9c669bac1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 01:31:29.904593 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:29.904558 6221 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 01:31:29.905002 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:29.904615 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q" podUID="73b85ea4-7469-459c-8ccd-3e9c669bac1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 01:31:30.904117 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:30.904082 6221 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 01:31:30.904348 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:30.904147 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q" podUID="73b85ea4-7469-459c-8ccd-3e9c669bac1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 01:31:32.139373 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:32.138925 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q" Oct 30 01:31:32.140039 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:32.140016 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q" Oct 30 01:31:32.141021 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:32.140997 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q" Oct 30 01:31:33.279637 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:33.279604 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" Oct 30 01:31:33.292019 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:33.291997 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" Oct 30 01:31:33.380809 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:33.380781 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" Oct 30 01:31:40.564808 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:40.564734 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65"] Oct 30 01:31:40.569697 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:40.569675 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65" Oct 30 01:31:40.599109 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:40.599080 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65"] Oct 30 01:31:40.614114 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:40.614087 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-istio-envoy\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65\" (UID: \"0dcc7bfb-9764-4e3e-9126-e45fa253a25e\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65" Oct 30 01:31:40.614259 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:40.614137 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-istiod-ca-cert\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65\" (UID: \"0dcc7bfb-9764-4e3e-9126-e45fa253a25e\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65" Oct 30 01:31:40.614259 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:40.614170 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-istio-data\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65\" (UID: \"0dcc7bfb-9764-4e3e-9126-e45fa253a25e\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65" Oct 30 01:31:40.614383 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:40.614296 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-istio-podinfo\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65\" (UID: \"0dcc7bfb-9764-4e3e-9126-e45fa253a25e\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65" Oct 30 01:31:40.614383 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:40.614324 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8ngm\" (UniqueName: \"kubernetes.io/projected/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-kube-api-access-x8ngm\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65\" (UID: \"0dcc7bfb-9764-4e3e-9126-e45fa253a25e\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65" Oct 30 01:31:40.614383 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:40.614353 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-workload-certs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65\" (UID: \"0dcc7bfb-9764-4e3e-9126-e45fa253a25e\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65" Oct 30 01:31:40.614563 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:40.614424 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-istio-token\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65\" (UID: \"0dcc7bfb-9764-4e3e-9126-e45fa253a25e\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65" Oct 30 01:31:40.614637 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:40.614596 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-workload-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65\" (UID: \"0dcc7bfb-9764-4e3e-9126-e45fa253a25e\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65" Oct 30 01:31:40.614715 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:40.614690 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-credential-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65\" (UID: \"0dcc7bfb-9764-4e3e-9126-e45fa253a25e\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65" Oct 30 01:31:40.715776 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:40.715742 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-credential-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65\" (UID: \"0dcc7bfb-9764-4e3e-9126-e45fa253a25e\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65" Oct 30 01:31:40.715959 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:40.715831 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-istio-envoy\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65\" (UID: \"0dcc7bfb-9764-4e3e-9126-e45fa253a25e\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65" Oct 30 01:31:40.715959 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:40.715869 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-istiod-ca-cert\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65\" (UID: \"0dcc7bfb-9764-4e3e-9126-e45fa253a25e\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65" Oct 30 01:31:40.715959 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:40.715898 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-istio-data\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65\" (UID: \"0dcc7bfb-9764-4e3e-9126-e45fa253a25e\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65" Oct 30 01:31:40.716373 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:40.716011 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-istio-podinfo\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65\" (UID: \"0dcc7bfb-9764-4e3e-9126-e45fa253a25e\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65" Oct 30 01:31:40.716373 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:40.716059 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8ngm\" (UniqueName: \"kubernetes.io/projected/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-kube-api-access-x8ngm\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65\" (UID: \"0dcc7bfb-9764-4e3e-9126-e45fa253a25e\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65" Oct 30 01:31:40.716373 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:40.716086 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-workload-certs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65\" (UID: \"0dcc7bfb-9764-4e3e-9126-e45fa253a25e\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65" Oct 30 01:31:40.716373 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:40.716125 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-istio-token\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65\" (UID: \"0dcc7bfb-9764-4e3e-9126-e45fa253a25e\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65" Oct 30 01:31:40.716373 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:40.716201 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-workload-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65\" (UID: \"0dcc7bfb-9764-4e3e-9126-e45fa253a25e\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65" Oct 30 01:31:40.716373 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:40.716204 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-credential-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65\" (UID: \"0dcc7bfb-9764-4e3e-9126-e45fa253a25e\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65" Oct 30 01:31:40.716746 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:40.716592 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-workload-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65\" (UID: \"0dcc7bfb-9764-4e3e-9126-e45fa253a25e\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65" Oct 30 01:31:40.717006 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:40.716983 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-istiod-ca-cert\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65\" (UID: \"0dcc7bfb-9764-4e3e-9126-e45fa253a25e\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65" Oct 30 01:31:40.717122 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:40.717077 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-workload-certs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65\" (UID: \"0dcc7bfb-9764-4e3e-9126-e45fa253a25e\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65" Oct 30 01:31:40.717122 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:40.717096 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-istio-data\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65\" (UID: \"0dcc7bfb-9764-4e3e-9126-e45fa253a25e\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65" Oct 30 01:31:40.718138 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:40.718113 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-istio-envoy\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65\" (UID: \"0dcc7bfb-9764-4e3e-9126-e45fa253a25e\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65" Oct 30 01:31:40.718734 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:40.718719 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-istio-podinfo\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65\" (UID: \"0dcc7bfb-9764-4e3e-9126-e45fa253a25e\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65" Oct 30 01:31:40.739507 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:40.739488 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-istio-token\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65\" (UID: \"0dcc7bfb-9764-4e3e-9126-e45fa253a25e\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65" Oct 30 01:31:40.741429 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:40.741410 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8ngm\" (UniqueName: \"kubernetes.io/projected/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-kube-api-access-x8ngm\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65\" (UID: \"0dcc7bfb-9764-4e3e-9126-e45fa253a25e\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65" Oct 30 01:31:40.901704 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:40.901624 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65" Oct 30 01:31:41.184737 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:41.184656 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h"] Oct 30 01:31:41.243756 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:41.243736 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65"] Oct 30 01:31:41.246601 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:31:41.246567 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dcc7bfb_9764_4e3e_9126_e45fa253a25e.slice/crio-40bfaa59ea7a764f2ef104c8fea59cf658ffd56b9d41f820687fe8d2c5b6b4c7 WatchSource:0}: Error finding container 40bfaa59ea7a764f2ef104c8fea59cf658ffd56b9d41f820687fe8d2c5b6b4c7: Status 404 returned error can't find the container with id 40bfaa59ea7a764f2ef104c8fea59cf658ffd56b9d41f820687fe8d2c5b6b4c7 Oct 30 01:31:41.248874 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:41.248844 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:31:41.248949 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:41.248902 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:31:41.248949 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:41.248933 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:31:41.463462 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:41.463365 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65" event={"ID":"0dcc7bfb-9764-4e3e-9126-e45fa253a25e","Type":"ContainerStarted","Data":"a2853aa6af3dc7068a99b2449661db1691446efcd8074b1e44da68d4b633fa15"} Oct 30 01:31:41.463462 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:41.463431 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65" event={"ID":"0dcc7bfb-9764-4e3e-9126-e45fa253a25e","Type":"ContainerStarted","Data":"40bfaa59ea7a764f2ef104c8fea59cf658ffd56b9d41f820687fe8d2c5b6b4c7"} Oct 30 01:31:41.521015 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:41.520961 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65" podStartSLOduration=1.520945474 podStartE2EDuration="1.520945474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:31:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:31:41.519031185 +0000 UTC m=+825.097318520" watchObservedRunningTime="2026-10-30 01:31:41.520945474 +0000 UTC m=+825.099232818" Oct 30 01:31:41.902501 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:41.902465 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65" Oct 30 01:31:42.904190 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:42.904153 6221 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 01:31:42.904679 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:42.904212 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65" podUID="0dcc7bfb-9764-4e3e-9126-e45fa253a25e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 01:31:43.902218 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:43.902182 6221 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 01:31:43.902439 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:43.902242 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65" podUID="0dcc7bfb-9764-4e3e-9126-e45fa253a25e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 01:31:44.902944 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:44.902902 6221 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 01:31:44.903375 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:44.902985 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65" podUID="0dcc7bfb-9764-4e3e-9126-e45fa253a25e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 01:31:45.902579 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:45.902544 6221 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 01:31:45.902742 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:45.902610 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65" podUID="0dcc7bfb-9764-4e3e-9126-e45fa253a25e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 01:31:46.902626 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:46.902588 6221 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 01:31:46.903027 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:46.902652 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65" podUID="0dcc7bfb-9764-4e3e-9126-e45fa253a25e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 01:31:48.230996 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:48.230960 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65" Oct 30 01:31:48.231894 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:48.231865 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65" Oct 30 01:31:48.234104 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:48.234072 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65" Oct 30 01:31:49.753856 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:49.753817 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:31:51.648829 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:31:51.648728 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:31:56.204630 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:56.204566 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h" podUID="e23cbbfb-1ff9-4233-8367-0bd6e3f642dd" containerName="main" containerID="cri-o://0460029c212eb6712651c7ea5cdc7d09d4605e9a34bbdc42dc82f2b16f6130de" gracePeriod=45 Oct 30 01:31:56.474808 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:56.474783 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h" Oct 30 01:31:56.553895 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:56.553862 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e23cbbfb-1ff9-4233-8367-0bd6e3f642dd-kube-api-access-d56jv\" (UniqueName: \"kubernetes.io/projected/e23cbbfb-1ff9-4233-8367-0bd6e3f642dd-kube-api-access-d56jv\") pod \"e23cbbfb-1ff9-4233-8367-0bd6e3f642dd\" (UID: \"e23cbbfb-1ff9-4233-8367-0bd6e3f642dd\") " Oct 30 01:31:56.554066 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:56.553964 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e23cbbfb-1ff9-4233-8367-0bd6e3f642dd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e23cbbfb-1ff9-4233-8367-0bd6e3f642dd-tmp-dir\") pod \"e23cbbfb-1ff9-4233-8367-0bd6e3f642dd\" (UID: \"e23cbbfb-1ff9-4233-8367-0bd6e3f642dd\") " Oct 30 01:31:56.554066 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:56.554009 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e23cbbfb-1ff9-4233-8367-0bd6e3f642dd-home\" (UniqueName: \"kubernetes.io/empty-dir/e23cbbfb-1ff9-4233-8367-0bd6e3f642dd-home\") pod \"e23cbbfb-1ff9-4233-8367-0bd6e3f642dd\" (UID: \"e23cbbfb-1ff9-4233-8367-0bd6e3f642dd\") " Oct 30 01:31:56.554066 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:56.554035 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e23cbbfb-1ff9-4233-8367-0bd6e3f642dd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e23cbbfb-1ff9-4233-8367-0bd6e3f642dd-model-cache\") pod \"e23cbbfb-1ff9-4233-8367-0bd6e3f642dd\" (UID: \"e23cbbfb-1ff9-4233-8367-0bd6e3f642dd\") " Oct 30 01:31:56.554066 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:56.554067 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e23cbbfb-1ff9-4233-8367-0bd6e3f642dd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e23cbbfb-1ff9-4233-8367-0bd6e3f642dd-dshm\") pod \"e23cbbfb-1ff9-4233-8367-0bd6e3f642dd\" (UID: \"e23cbbfb-1ff9-4233-8367-0bd6e3f642dd\") " Oct 30 01:31:56.554305 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:56.554110 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e23cbbfb-1ff9-4233-8367-0bd6e3f642dd-tls-certs\" (UniqueName: \"kubernetes.io/secret/e23cbbfb-1ff9-4233-8367-0bd6e3f642dd-tls-certs\") pod \"e23cbbfb-1ff9-4233-8367-0bd6e3f642dd\" (UID: \"e23cbbfb-1ff9-4233-8367-0bd6e3f642dd\") " Oct 30 01:31:56.554305 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:56.554246 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e23cbbfb-1ff9-4233-8367-0bd6e3f642dd-tmp-dir" pod "e23cbbfb-1ff9-4233-8367-0bd6e3f642dd" (UID: "e23cbbfb-1ff9-4233-8367-0bd6e3f642dd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:31:56.554305 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:56.554294 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e23cbbfb-1ff9-4233-8367-0bd6e3f642dd-model-cache" pod "e23cbbfb-1ff9-4233-8367-0bd6e3f642dd" (UID: "e23cbbfb-1ff9-4233-8367-0bd6e3f642dd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:31:56.554483 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:56.554307 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e23cbbfb-1ff9-4233-8367-0bd6e3f642dd-home" pod "e23cbbfb-1ff9-4233-8367-0bd6e3f642dd" (UID: "e23cbbfb-1ff9-4233-8367-0bd6e3f642dd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:31:56.554571 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:56.554505 6221 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e23cbbfb-1ff9-4233-8367-0bd6e3f642dd-tmp-dir\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:31:56.554571 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:56.554520 6221 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e23cbbfb-1ff9-4233-8367-0bd6e3f642dd-home\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:31:56.554571 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:56.554557 6221 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e23cbbfb-1ff9-4233-8367-0bd6e3f642dd-model-cache\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:31:56.556108 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:56.556085 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e23cbbfb-1ff9-4233-8367-0bd6e3f642dd-kube-api-access-d56jv" pod "e23cbbfb-1ff9-4233-8367-0bd6e3f642dd" (UID: "e23cbbfb-1ff9-4233-8367-0bd6e3f642dd"). InnerVolumeSpecName "kube-api-access-d56jv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:31:56.556108 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:56.556096 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e23cbbfb-1ff9-4233-8367-0bd6e3f642dd-dshm" pod "e23cbbfb-1ff9-4233-8367-0bd6e3f642dd" (UID: "e23cbbfb-1ff9-4233-8367-0bd6e3f642dd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:31:56.556254 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:56.556122 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e23cbbfb-1ff9-4233-8367-0bd6e3f642dd-tls-certs" pod "e23cbbfb-1ff9-4233-8367-0bd6e3f642dd" (UID: "e23cbbfb-1ff9-4233-8367-0bd6e3f642dd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:31:56.574195 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:56.574168 6221 generic.go:356] "Generic (PLEG): container finished" podID="e23cbbfb-1ff9-4233-8367-0bd6e3f642dd" containerID="0460029c212eb6712651c7ea5cdc7d09d4605e9a34bbdc42dc82f2b16f6130de" exitCode=0 Oct 30 01:31:56.574310 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:56.574219 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h" Oct 30 01:31:56.574310 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:56.574243 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h" event={"ID":"e23cbbfb-1ff9-4233-8367-0bd6e3f642dd","Type":"ContainerDied","Data":"0460029c212eb6712651c7ea5cdc7d09d4605e9a34bbdc42dc82f2b16f6130de"} Oct 30 01:31:56.574310 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:56.574284 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h" event={"ID":"e23cbbfb-1ff9-4233-8367-0bd6e3f642dd","Type":"ContainerDied","Data":"e47f116ec88776e54fdbf93d36c89589dde6564df78323b9584fe036985fc504"} Oct 30 01:31:56.574310 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:56.574304 6221 scope.go:122] "RemoveContainer" containerID="0460029c212eb6712651c7ea5cdc7d09d4605e9a34bbdc42dc82f2b16f6130de" Oct 30 01:31:56.584685 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:56.584662 6221 scope.go:122] "RemoveContainer" containerID="0460029c212eb6712651c7ea5cdc7d09d4605e9a34bbdc42dc82f2b16f6130de" Oct 30 01:31:56.584952 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:31:56.584932 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0460029c212eb6712651c7ea5cdc7d09d4605e9a34bbdc42dc82f2b16f6130de\": container with ID starting with 0460029c212eb6712651c7ea5cdc7d09d4605e9a34bbdc42dc82f2b16f6130de not found: ID does not exist" containerID="0460029c212eb6712651c7ea5cdc7d09d4605e9a34bbdc42dc82f2b16f6130de" Oct 30 01:31:56.585000 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:56.584961 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0460029c212eb6712651c7ea5cdc7d09d4605e9a34bbdc42dc82f2b16f6130de"} err="failed to get container status \"0460029c212eb6712651c7ea5cdc7d09d4605e9a34bbdc42dc82f2b16f6130de\": rpc error: code = NotFound desc = could not find container \"0460029c212eb6712651c7ea5cdc7d09d4605e9a34bbdc42dc82f2b16f6130de\": container with ID starting with 0460029c212eb6712651c7ea5cdc7d09d4605e9a34bbdc42dc82f2b16f6130de not found: ID does not exist" Oct 30 01:31:56.626330 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:56.626298 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h"] Oct 30 01:31:56.628612 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:56.628590 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-cf82f8/e2e-conflict-a-d3477dc4-kserve-656cb4dbc9-x495h"] Oct 30 01:31:56.655589 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:56.655560 6221 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e23cbbfb-1ff9-4233-8367-0bd6e3f642dd-dshm\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:31:56.655589 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:56.655585 6221 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e23cbbfb-1ff9-4233-8367-0bd6e3f642dd-tls-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:31:56.655589 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:56.655596 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d56jv\" (UniqueName: \"kubernetes.io/projected/e23cbbfb-1ff9-4233-8367-0bd6e3f642dd-kube-api-access-d56jv\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:31:57.199465 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:57.199422 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx"] Oct 30 01:31:57.200159 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:57.200128 6221 cpu_manager.go:414] "containerMap: removing container" podUID="e23cbbfb-1ff9-4233-8367-0bd6e3f642dd" containerName="main" Oct 30 01:31:57.200159 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:57.200154 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e23cbbfb-1ff9-4233-8367-0bd6e3f642dd" containerName="main" Oct 30 01:31:57.200359 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:57.200305 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="e23cbbfb-1ff9-4233-8367-0bd6e3f642dd" containerName="main" Oct 30 01:31:57.232693 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:57.232656 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx"] Oct 30 01:31:57.233172 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:57.232796 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx" Oct 30 01:31:57.272839 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:57.272808 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/324f43dc-720f-4b56-9f26-39ffce236212-istiod-ca-cert\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-vhvmx\" (UID: \"324f43dc-720f-4b56-9f26-39ffce236212\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx" Oct 30 01:31:57.272839 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:57.272840 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/324f43dc-720f-4b56-9f26-39ffce236212-istio-token\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-vhvmx\" (UID: \"324f43dc-720f-4b56-9f26-39ffce236212\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx" Oct 30 01:31:57.273076 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:57.272861 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z88h9\" (UniqueName: \"kubernetes.io/projected/324f43dc-720f-4b56-9f26-39ffce236212-kube-api-access-z88h9\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-vhvmx\" (UID: \"324f43dc-720f-4b56-9f26-39ffce236212\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx" Oct 30 01:31:57.273076 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:57.273029 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/324f43dc-720f-4b56-9f26-39ffce236212-workload-certs\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-vhvmx\" (UID: \"324f43dc-720f-4b56-9f26-39ffce236212\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx" Oct 30 01:31:57.273164 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:57.273114 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/324f43dc-720f-4b56-9f26-39ffce236212-istio-envoy\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-vhvmx\" (UID: \"324f43dc-720f-4b56-9f26-39ffce236212\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx" Oct 30 01:31:57.273212 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:57.273155 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/324f43dc-720f-4b56-9f26-39ffce236212-istio-data\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-vhvmx\" (UID: \"324f43dc-720f-4b56-9f26-39ffce236212\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx" Oct 30 01:31:57.273285 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:57.273266 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/324f43dc-720f-4b56-9f26-39ffce236212-istio-podinfo\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-vhvmx\" (UID: \"324f43dc-720f-4b56-9f26-39ffce236212\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx" Oct 30 01:31:57.273401 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:57.273368 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/324f43dc-720f-4b56-9f26-39ffce236212-workload-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-vhvmx\" (UID: \"324f43dc-720f-4b56-9f26-39ffce236212\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx" Oct 30 01:31:57.273505 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:57.273451 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/324f43dc-720f-4b56-9f26-39ffce236212-credential-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-vhvmx\" (UID: \"324f43dc-720f-4b56-9f26-39ffce236212\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx" Oct 30 01:31:57.374814 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:57.374785 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z88h9\" (UniqueName: \"kubernetes.io/projected/324f43dc-720f-4b56-9f26-39ffce236212-kube-api-access-z88h9\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-vhvmx\" (UID: \"324f43dc-720f-4b56-9f26-39ffce236212\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx" Oct 30 01:31:57.374993 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:57.374834 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/324f43dc-720f-4b56-9f26-39ffce236212-workload-certs\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-vhvmx\" (UID: \"324f43dc-720f-4b56-9f26-39ffce236212\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx" Oct 30 01:31:57.374993 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:57.374854 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/324f43dc-720f-4b56-9f26-39ffce236212-istio-envoy\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-vhvmx\" (UID: \"324f43dc-720f-4b56-9f26-39ffce236212\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx" Oct 30 01:31:57.374993 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:57.374874 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/324f43dc-720f-4b56-9f26-39ffce236212-istio-data\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-vhvmx\" (UID: \"324f43dc-720f-4b56-9f26-39ffce236212\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx" Oct 30 01:31:57.374993 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:57.374959 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/324f43dc-720f-4b56-9f26-39ffce236212-istio-podinfo\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-vhvmx\" (UID: \"324f43dc-720f-4b56-9f26-39ffce236212\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx" Oct 30 01:31:57.375228 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:57.375006 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/324f43dc-720f-4b56-9f26-39ffce236212-workload-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-vhvmx\" (UID: \"324f43dc-720f-4b56-9f26-39ffce236212\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx" Oct 30 01:31:57.375228 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:57.375042 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/324f43dc-720f-4b56-9f26-39ffce236212-credential-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-vhvmx\" (UID: \"324f43dc-720f-4b56-9f26-39ffce236212\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx" Oct 30 01:31:57.375228 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:57.375156 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/324f43dc-720f-4b56-9f26-39ffce236212-istiod-ca-cert\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-vhvmx\" (UID: \"324f43dc-720f-4b56-9f26-39ffce236212\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx" Oct 30 01:31:57.375228 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:57.375175 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/324f43dc-720f-4b56-9f26-39ffce236212-istio-token\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-vhvmx\" (UID: \"324f43dc-720f-4b56-9f26-39ffce236212\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx" Oct 30 01:31:57.375709 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:57.375684 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/324f43dc-720f-4b56-9f26-39ffce236212-workload-certs\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-vhvmx\" (UID: \"324f43dc-720f-4b56-9f26-39ffce236212\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx" Oct 30 01:31:57.375862 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:57.375835 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/324f43dc-720f-4b56-9f26-39ffce236212-istio-data\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-vhvmx\" (UID: \"324f43dc-720f-4b56-9f26-39ffce236212\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx" Oct 30 01:31:57.375934 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:57.375875 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/324f43dc-720f-4b56-9f26-39ffce236212-workload-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-vhvmx\" (UID: \"324f43dc-720f-4b56-9f26-39ffce236212\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx" Oct 30 01:31:57.375934 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:57.375922 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/324f43dc-720f-4b56-9f26-39ffce236212-credential-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-vhvmx\" (UID: \"324f43dc-720f-4b56-9f26-39ffce236212\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx" Oct 30 01:31:57.376175 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:57.376152 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/324f43dc-720f-4b56-9f26-39ffce236212-istiod-ca-cert\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-vhvmx\" (UID: \"324f43dc-720f-4b56-9f26-39ffce236212\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx" Oct 30 01:31:57.377540 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:57.377502 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/324f43dc-720f-4b56-9f26-39ffce236212-istio-podinfo\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-vhvmx\" (UID: \"324f43dc-720f-4b56-9f26-39ffce236212\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx" Oct 30 01:31:57.377641 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:57.377507 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/324f43dc-720f-4b56-9f26-39ffce236212-istio-envoy\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-vhvmx\" (UID: \"324f43dc-720f-4b56-9f26-39ffce236212\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx" Oct 30 01:31:57.394864 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:57.394839 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z88h9\" (UniqueName: \"kubernetes.io/projected/324f43dc-720f-4b56-9f26-39ffce236212-kube-api-access-z88h9\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-vhvmx\" (UID: \"324f43dc-720f-4b56-9f26-39ffce236212\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx" Oct 30 01:31:57.395317 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:57.395295 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/324f43dc-720f-4b56-9f26-39ffce236212-istio-token\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-vhvmx\" (UID: \"324f43dc-720f-4b56-9f26-39ffce236212\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx" Oct 30 01:31:57.552515 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:57.552485 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx" Oct 30 01:31:57.870679 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:57.870598 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx"] Oct 30 01:31:57.874027 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:31:57.873998 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod324f43dc_720f_4b56_9f26_39ffce236212.slice/crio-d1a9147c028e0541a000372997f4fbad13762e8a9bc7c3c730db1e43ad43568b WatchSource:0}: Error finding container d1a9147c028e0541a000372997f4fbad13762e8a9bc7c3c730db1e43ad43568b: Status 404 returned error can't find the container with id d1a9147c028e0541a000372997f4fbad13762e8a9bc7c3c730db1e43ad43568b Oct 30 01:31:57.876446 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:57.876417 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:31:57.876502 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:57.876478 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:31:57.876563 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:57.876507 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:31:58.547696 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:58.547663 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e23cbbfb-1ff9-4233-8367-0bd6e3f642dd" path="/var/lib/kubelet/pods/e23cbbfb-1ff9-4233-8367-0bd6e3f642dd/volumes" Oct 30 01:31:58.588627 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:58.588574 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx" event={"ID":"324f43dc-720f-4b56-9f26-39ffce236212","Type":"ContainerStarted","Data":"2b26eec3041b65c7b139e454733faeed578314ece02914c03b4cffad09e947b2"} Oct 30 01:31:58.588834 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:58.588648 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx" event={"ID":"324f43dc-720f-4b56-9f26-39ffce236212","Type":"ContainerStarted","Data":"d1a9147c028e0541a000372997f4fbad13762e8a9bc7c3c730db1e43ad43568b"} Oct 30 01:31:58.656175 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:58.656120 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx" podStartSLOduration=1.656106432 podStartE2EDuration="1.656106432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:31:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:31:58.655647176 +0000 UTC m=+842.233934514" watchObservedRunningTime="2026-10-30 01:31:58.656106432 +0000 UTC m=+842.234393767" Oct 30 01:31:59.554644 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:59.554601 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx" Oct 30 01:31:59.556623 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:59.556595 6221 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 01:31:59.556779 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:31:59.556688 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx" podUID="324f43dc-720f-4b56-9f26-39ffce236212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 01:32:00.553658 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:00.553620 6221 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 01:32:00.553834 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:00.553696 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx" podUID="324f43dc-720f-4b56-9f26-39ffce236212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 01:32:01.442648 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:01.442597 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t"] Oct 30 01:32:01.451468 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:01.450058 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" Oct 30 01:32:01.464796 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:01.464768 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t"] Oct 30 01:32:01.472739 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:01.472714 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-cf82f8\"/\"openshift-service-ca.crt\"" Oct 30 01:32:01.472739 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:01.472738 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-cf82f8\"/\"default-dockercfg-d6pv6\"" Oct 30 01:32:01.477583 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:01.477566 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-cf82f8\"/\"distinct-w1-cf82f8-kserve-self-signed-certs\"" Oct 30 01:32:01.477668 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:01.477593 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-cf82f8\"/\"kube-root-ca.crt\"" Oct 30 01:32:01.553650 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:01.553619 6221 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 01:32:01.553844 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:01.553677 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx" podUID="324f43dc-720f-4b56-9f26-39ffce236212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 01:32:01.567023 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:01.566987 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b3c0e08-1371-4184-816a-d8b501526190-model-cache\") pod \"distinct-w1-cf82f8-kserve-c789bf4f-p9s8t\" (UID: \"3b3c0e08-1371-4184-816a-d8b501526190\") " pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" Oct 30 01:32:01.567185 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:01.567048 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b3c0e08-1371-4184-816a-d8b501526190-home\") pod \"distinct-w1-cf82f8-kserve-c789bf4f-p9s8t\" (UID: \"3b3c0e08-1371-4184-816a-d8b501526190\") " pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" Oct 30 01:32:01.567185 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:01.567101 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4s49\" (UniqueName: \"kubernetes.io/projected/3b3c0e08-1371-4184-816a-d8b501526190-kube-api-access-l4s49\") pod \"distinct-w1-cf82f8-kserve-c789bf4f-p9s8t\" (UID: \"3b3c0e08-1371-4184-816a-d8b501526190\") " pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" Oct 30 01:32:01.567257 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:01.567215 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b3c0e08-1371-4184-816a-d8b501526190-tls-certs\") pod \"distinct-w1-cf82f8-kserve-c789bf4f-p9s8t\" (UID: \"3b3c0e08-1371-4184-816a-d8b501526190\") " pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" Oct 30 01:32:01.567295 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:01.567276 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b3c0e08-1371-4184-816a-d8b501526190-dshm\") pod \"distinct-w1-cf82f8-kserve-c789bf4f-p9s8t\" (UID: \"3b3c0e08-1371-4184-816a-d8b501526190\") " pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" Oct 30 01:32:01.567376 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:01.567356 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b3c0e08-1371-4184-816a-d8b501526190-tmp-dir\") pod \"distinct-w1-cf82f8-kserve-c789bf4f-p9s8t\" (UID: \"3b3c0e08-1371-4184-816a-d8b501526190\") " pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" Oct 30 01:32:01.680557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:01.675437 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b3c0e08-1371-4184-816a-d8b501526190-model-cache\") pod \"distinct-w1-cf82f8-kserve-c789bf4f-p9s8t\" (UID: \"3b3c0e08-1371-4184-816a-d8b501526190\") " pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" Oct 30 01:32:01.680557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:01.675495 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b3c0e08-1371-4184-816a-d8b501526190-home\") pod \"distinct-w1-cf82f8-kserve-c789bf4f-p9s8t\" (UID: \"3b3c0e08-1371-4184-816a-d8b501526190\") " pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" Oct 30 01:32:01.680557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:01.675524 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4s49\" (UniqueName: \"kubernetes.io/projected/3b3c0e08-1371-4184-816a-d8b501526190-kube-api-access-l4s49\") pod \"distinct-w1-cf82f8-kserve-c789bf4f-p9s8t\" (UID: \"3b3c0e08-1371-4184-816a-d8b501526190\") " pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" Oct 30 01:32:01.680557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:01.675596 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b3c0e08-1371-4184-816a-d8b501526190-tls-certs\") pod \"distinct-w1-cf82f8-kserve-c789bf4f-p9s8t\" (UID: \"3b3c0e08-1371-4184-816a-d8b501526190\") " pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" Oct 30 01:32:01.680557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:01.675659 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b3c0e08-1371-4184-816a-d8b501526190-dshm\") pod \"distinct-w1-cf82f8-kserve-c789bf4f-p9s8t\" (UID: \"3b3c0e08-1371-4184-816a-d8b501526190\") " pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" Oct 30 01:32:01.680557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:01.675710 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b3c0e08-1371-4184-816a-d8b501526190-tmp-dir\") pod \"distinct-w1-cf82f8-kserve-c789bf4f-p9s8t\" (UID: \"3b3c0e08-1371-4184-816a-d8b501526190\") " pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" Oct 30 01:32:01.680557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:01.676195 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b3c0e08-1371-4184-816a-d8b501526190-tmp-dir\") pod \"distinct-w1-cf82f8-kserve-c789bf4f-p9s8t\" (UID: \"3b3c0e08-1371-4184-816a-d8b501526190\") " pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" Oct 30 01:32:01.680557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:01.676449 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b3c0e08-1371-4184-816a-d8b501526190-model-cache\") pod \"distinct-w1-cf82f8-kserve-c789bf4f-p9s8t\" (UID: \"3b3c0e08-1371-4184-816a-d8b501526190\") " pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" Oct 30 01:32:01.680557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:01.676714 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b3c0e08-1371-4184-816a-d8b501526190-home\") pod \"distinct-w1-cf82f8-kserve-c789bf4f-p9s8t\" (UID: \"3b3c0e08-1371-4184-816a-d8b501526190\") " pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" Oct 30 01:32:01.686702 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:01.683494 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b3c0e08-1371-4184-816a-d8b501526190-tls-certs\") pod \"distinct-w1-cf82f8-kserve-c789bf4f-p9s8t\" (UID: \"3b3c0e08-1371-4184-816a-d8b501526190\") " pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" Oct 30 01:32:01.697675 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:01.687346 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b3c0e08-1371-4184-816a-d8b501526190-dshm\") pod \"distinct-w1-cf82f8-kserve-c789bf4f-p9s8t\" (UID: \"3b3c0e08-1371-4184-816a-d8b501526190\") " pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" Oct 30 01:32:01.709432 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:01.709331 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4s49\" (UniqueName: \"kubernetes.io/projected/3b3c0e08-1371-4184-816a-d8b501526190-kube-api-access-l4s49\") pod \"distinct-w1-cf82f8-kserve-c789bf4f-p9s8t\" (UID: \"3b3c0e08-1371-4184-816a-d8b501526190\") " pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" Oct 30 01:32:01.770554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:01.770358 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" Oct 30 01:32:02.092921 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:02.092885 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t"] Oct 30 01:32:02.097055 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:32:02.097024 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b3c0e08_1371_4184_816a_d8b501526190.slice/crio-8a60f7b53f0197c80d0a7429af1f726e2d870220457e850c289578735cdb3f09 WatchSource:0}: Error finding container 8a60f7b53f0197c80d0a7429af1f726e2d870220457e850c289578735cdb3f09: Status 404 returned error can't find the container with id 8a60f7b53f0197c80d0a7429af1f726e2d870220457e850c289578735cdb3f09 Oct 30 01:32:02.553669 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:02.553635 6221 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 01:32:02.554074 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:02.553691 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx" podUID="324f43dc-720f-4b56-9f26-39ffce236212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 01:32:02.634950 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:02.634910 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" event={"ID":"3b3c0e08-1371-4184-816a-d8b501526190","Type":"ContainerStarted","Data":"ab134ba62aec4bfcd7016f63343e4f5e2608ea01be89a49e2b74c24aeea0baaa"} Oct 30 01:32:02.635168 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:02.634962 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" event={"ID":"3b3c0e08-1371-4184-816a-d8b501526190","Type":"ContainerStarted","Data":"8a60f7b53f0197c80d0a7429af1f726e2d870220457e850c289578735cdb3f09"} Oct 30 01:32:02.709390 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:02.709326 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" podStartSLOduration=1.709308281 podStartE2EDuration="1.709308281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:02.703941083 +0000 UTC m=+846.282228423" watchObservedRunningTime="2026-10-30 01:32:02.709308281 +0000 UTC m=+846.287595620" Oct 30 01:32:02.771515 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:02.771479 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" Oct 30 01:32:03.553586 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:03.553516 6221 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 01:32:03.553776 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:03.553602 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx" podUID="324f43dc-720f-4b56-9f26-39ffce236212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 01:32:04.552850 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:04.552809 6221 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 01:32:04.553043 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:04.552889 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx" podUID="324f43dc-720f-4b56-9f26-39ffce236212" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 01:32:05.557261 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:05.557229 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx" Oct 30 01:32:05.557857 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:05.557503 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx" Oct 30 01:32:05.558366 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:05.558346 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx" Oct 30 01:32:09.028458 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:09.028414 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:32:11.771970 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:11.771941 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" Oct 30 01:32:11.785451 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:11.785421 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" Oct 30 01:32:12.729117 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:12.729088 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" Oct 30 01:32:17.959684 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:17.959644 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq"] Oct 30 01:32:17.963323 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:17.963302 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq" Oct 30 01:32:17.978962 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:17.978943 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-a3a2ac-openshift-default-dockercfg-mwwtj\"" Oct 30 01:32:17.994370 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:17.994341 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq"] Oct 30 01:32:18.080583 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:18.080525 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6858a801-456a-4716-be05-8a3a7bd80334-istio-token\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq\" (UID: \"6858a801-456a-4716-be05-8a3a7bd80334\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq" Oct 30 01:32:18.080756 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:18.080595 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6858a801-456a-4716-be05-8a3a7bd80334-istio-data\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq\" (UID: \"6858a801-456a-4716-be05-8a3a7bd80334\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq" Oct 30 01:32:18.080756 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:18.080662 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwn7g\" (UniqueName: \"kubernetes.io/projected/6858a801-456a-4716-be05-8a3a7bd80334-kube-api-access-zwn7g\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq\" (UID: \"6858a801-456a-4716-be05-8a3a7bd80334\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq" Oct 30 01:32:18.080756 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:18.080698 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6858a801-456a-4716-be05-8a3a7bd80334-istio-envoy\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq\" (UID: \"6858a801-456a-4716-be05-8a3a7bd80334\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq" Oct 30 01:32:18.080756 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:18.080720 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6858a801-456a-4716-be05-8a3a7bd80334-workload-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq\" (UID: \"6858a801-456a-4716-be05-8a3a7bd80334\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq" Oct 30 01:32:18.080890 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:18.080822 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6858a801-456a-4716-be05-8a3a7bd80334-istiod-ca-cert\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq\" (UID: \"6858a801-456a-4716-be05-8a3a7bd80334\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq" Oct 30 01:32:18.080890 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:18.080865 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6858a801-456a-4716-be05-8a3a7bd80334-istio-podinfo\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq\" (UID: \"6858a801-456a-4716-be05-8a3a7bd80334\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq" Oct 30 01:32:18.080952 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:18.080911 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6858a801-456a-4716-be05-8a3a7bd80334-credential-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq\" (UID: \"6858a801-456a-4716-be05-8a3a7bd80334\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq" Oct 30 01:32:18.080952 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:18.080937 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6858a801-456a-4716-be05-8a3a7bd80334-workload-certs\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq\" (UID: \"6858a801-456a-4716-be05-8a3a7bd80334\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq" Oct 30 01:32:18.181773 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:18.181739 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6858a801-456a-4716-be05-8a3a7bd80334-istio-token\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq\" (UID: \"6858a801-456a-4716-be05-8a3a7bd80334\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq" Oct 30 01:32:18.181773 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:18.181773 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6858a801-456a-4716-be05-8a3a7bd80334-istio-data\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq\" (UID: \"6858a801-456a-4716-be05-8a3a7bd80334\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq" Oct 30 01:32:18.182005 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:18.181802 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwn7g\" (UniqueName: \"kubernetes.io/projected/6858a801-456a-4716-be05-8a3a7bd80334-kube-api-access-zwn7g\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq\" (UID: \"6858a801-456a-4716-be05-8a3a7bd80334\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq" Oct 30 01:32:18.182005 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:18.181835 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6858a801-456a-4716-be05-8a3a7bd80334-istio-envoy\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq\" (UID: \"6858a801-456a-4716-be05-8a3a7bd80334\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq" Oct 30 01:32:18.182005 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:18.181855 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6858a801-456a-4716-be05-8a3a7bd80334-workload-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq\" (UID: \"6858a801-456a-4716-be05-8a3a7bd80334\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq" Oct 30 01:32:18.182005 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:18.181878 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6858a801-456a-4716-be05-8a3a7bd80334-istiod-ca-cert\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq\" (UID: \"6858a801-456a-4716-be05-8a3a7bd80334\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq" Oct 30 01:32:18.182005 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:18.181905 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6858a801-456a-4716-be05-8a3a7bd80334-istio-podinfo\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq\" (UID: \"6858a801-456a-4716-be05-8a3a7bd80334\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq" Oct 30 01:32:18.182005 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:18.181946 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6858a801-456a-4716-be05-8a3a7bd80334-credential-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq\" (UID: \"6858a801-456a-4716-be05-8a3a7bd80334\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq" Oct 30 01:32:18.182005 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:18.181983 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6858a801-456a-4716-be05-8a3a7bd80334-workload-certs\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq\" (UID: \"6858a801-456a-4716-be05-8a3a7bd80334\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq" Oct 30 01:32:18.182378 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:18.182341 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6858a801-456a-4716-be05-8a3a7bd80334-istio-data\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq\" (UID: \"6858a801-456a-4716-be05-8a3a7bd80334\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq" Oct 30 01:32:18.182442 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:18.182343 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6858a801-456a-4716-be05-8a3a7bd80334-workload-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq\" (UID: \"6858a801-456a-4716-be05-8a3a7bd80334\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq" Oct 30 01:32:18.182442 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:18.182404 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6858a801-456a-4716-be05-8a3a7bd80334-workload-certs\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq\" (UID: \"6858a801-456a-4716-be05-8a3a7bd80334\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq" Oct 30 01:32:18.182601 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:18.182444 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6858a801-456a-4716-be05-8a3a7bd80334-credential-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq\" (UID: \"6858a801-456a-4716-be05-8a3a7bd80334\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq" Oct 30 01:32:18.182785 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:18.182761 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6858a801-456a-4716-be05-8a3a7bd80334-istiod-ca-cert\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq\" (UID: \"6858a801-456a-4716-be05-8a3a7bd80334\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq" Oct 30 01:32:18.184115 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:18.184097 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6858a801-456a-4716-be05-8a3a7bd80334-istio-envoy\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq\" (UID: \"6858a801-456a-4716-be05-8a3a7bd80334\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq" Oct 30 01:32:18.184414 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:18.184392 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6858a801-456a-4716-be05-8a3a7bd80334-istio-podinfo\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq\" (UID: \"6858a801-456a-4716-be05-8a3a7bd80334\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq" Oct 30 01:32:18.201230 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:18.201209 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6858a801-456a-4716-be05-8a3a7bd80334-istio-token\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq\" (UID: \"6858a801-456a-4716-be05-8a3a7bd80334\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq" Oct 30 01:32:18.201343 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:18.201326 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwn7g\" (UniqueName: \"kubernetes.io/projected/6858a801-456a-4716-be05-8a3a7bd80334-kube-api-access-zwn7g\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq\" (UID: \"6858a801-456a-4716-be05-8a3a7bd80334\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq" Oct 30 01:32:18.277055 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:18.276989 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq" Oct 30 01:32:18.594961 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:18.594926 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq"] Oct 30 01:32:18.598153 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:32:18.598120 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6858a801_456a_4716_be05_8a3a7bd80334.slice/crio-18da2a2704e18862fa7d1c18a5f1142485e647d6c6f18b959e90b5ee1daaa89a WatchSource:0}: Error finding container 18da2a2704e18862fa7d1c18a5f1142485e647d6c6f18b959e90b5ee1daaa89a: Status 404 returned error can't find the container with id 18da2a2704e18862fa7d1c18a5f1142485e647d6c6f18b959e90b5ee1daaa89a Oct 30 01:32:18.600458 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:18.600426 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:32:18.600563 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:18.600489 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:32:18.600563 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:18.600521 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:32:18.753649 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:18.753610 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq" event={"ID":"6858a801-456a-4716-be05-8a3a7bd80334","Type":"ContainerStarted","Data":"3ab22f87b33c77d11f6f9ded0e2b60d140bdfb674713ad874369a98eaa5a0d72"} Oct 30 01:32:18.753831 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:18.753675 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq" event={"ID":"6858a801-456a-4716-be05-8a3a7bd80334","Type":"ContainerStarted","Data":"18da2a2704e18862fa7d1c18a5f1142485e647d6c6f18b959e90b5ee1daaa89a"} Oct 30 01:32:18.798295 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:18.798249 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq" podStartSLOduration=1.798236625 podStartE2EDuration="1.798236625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:18.797611125 +0000 UTC m=+862.375898461" watchObservedRunningTime="2026-10-30 01:32:18.798236625 +0000 UTC m=+862.376523961" Oct 30 01:32:19.278016 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:19.277976 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq" Oct 30 01:32:19.279677 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:19.279655 6221 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 01:32:19.279737 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:19.279702 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq" podUID="6858a801-456a-4716-be05-8a3a7bd80334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 01:32:20.277983 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:20.277947 6221 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 01:32:20.278172 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:20.278004 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq" podUID="6858a801-456a-4716-be05-8a3a7bd80334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 01:32:21.278303 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:21.278267 6221 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 01:32:21.278730 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:21.278318 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq" podUID="6858a801-456a-4716-be05-8a3a7bd80334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 01:32:21.495202 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:32:21.495168 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:32:22.277679 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:22.277640 6221 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 01:32:22.277880 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:22.277698 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq" podUID="6858a801-456a-4716-be05-8a3a7bd80334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 01:32:23.277671 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:23.277635 6221 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 01:32:23.278083 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:23.277685 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq" podUID="6858a801-456a-4716-be05-8a3a7bd80334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 01:32:24.277487 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:24.277447 6221 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 01:32:24.277707 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:24.277500 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq" podUID="6858a801-456a-4716-be05-8a3a7bd80334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 01:32:25.550232 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:25.550197 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq" Oct 30 01:32:25.551217 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:25.551190 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq" Oct 30 01:32:25.551951 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:25.551894 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq" Oct 30 01:32:25.695984 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:25.695949 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq"] Oct 30 01:32:25.700806 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:25.700780 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" Oct 30 01:32:25.717808 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:25.717778 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-cf82f8\"/\"distinct-2-w1-cf82f8-kserve-self-signed-certs\"" Oct 30 01:32:25.718692 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:25.718669 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq"] Oct 30 01:32:25.804583 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:25.804465 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9df52a01-0c8d-44ba-9fd7-9cb2fa41947a-tmp-dir\") pod \"distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq\" (UID: \"9df52a01-0c8d-44ba-9fd7-9cb2fa41947a\") " pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" Oct 30 01:32:25.804583 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:25.804505 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9df52a01-0c8d-44ba-9fd7-9cb2fa41947a-home\") pod \"distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq\" (UID: \"9df52a01-0c8d-44ba-9fd7-9cb2fa41947a\") " pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" Oct 30 01:32:25.804583 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:25.804541 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9df52a01-0c8d-44ba-9fd7-9cb2fa41947a-dshm\") pod \"distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq\" (UID: \"9df52a01-0c8d-44ba-9fd7-9cb2fa41947a\") " pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" Oct 30 01:32:25.804791 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:25.804658 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9df52a01-0c8d-44ba-9fd7-9cb2fa41947a-model-cache\") pod \"distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq\" (UID: \"9df52a01-0c8d-44ba-9fd7-9cb2fa41947a\") " pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" Oct 30 01:32:25.804791 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:25.804676 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr87s\" (UniqueName: \"kubernetes.io/projected/9df52a01-0c8d-44ba-9fd7-9cb2fa41947a-kube-api-access-dr87s\") pod \"distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq\" (UID: \"9df52a01-0c8d-44ba-9fd7-9cb2fa41947a\") " pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" Oct 30 01:32:25.804850 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:25.804808 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9df52a01-0c8d-44ba-9fd7-9cb2fa41947a-tls-certs\") pod \"distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq\" (UID: \"9df52a01-0c8d-44ba-9fd7-9cb2fa41947a\") " pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" Oct 30 01:32:25.905814 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:25.905773 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9df52a01-0c8d-44ba-9fd7-9cb2fa41947a-tmp-dir\") pod \"distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq\" (UID: \"9df52a01-0c8d-44ba-9fd7-9cb2fa41947a\") " pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" Oct 30 01:32:25.907450 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:25.907421 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9df52a01-0c8d-44ba-9fd7-9cb2fa41947a-home\") pod \"distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq\" (UID: \"9df52a01-0c8d-44ba-9fd7-9cb2fa41947a\") " pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" Oct 30 01:32:25.907671 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:25.907653 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9df52a01-0c8d-44ba-9fd7-9cb2fa41947a-dshm\") pod \"distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq\" (UID: \"9df52a01-0c8d-44ba-9fd7-9cb2fa41947a\") " pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" Oct 30 01:32:25.908405 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:25.908386 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9df52a01-0c8d-44ba-9fd7-9cb2fa41947a-model-cache\") pod \"distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq\" (UID: \"9df52a01-0c8d-44ba-9fd7-9cb2fa41947a\") " pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" Oct 30 01:32:25.908847 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:25.908829 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr87s\" (UniqueName: \"kubernetes.io/projected/9df52a01-0c8d-44ba-9fd7-9cb2fa41947a-kube-api-access-dr87s\") pod \"distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq\" (UID: \"9df52a01-0c8d-44ba-9fd7-9cb2fa41947a\") " pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" Oct 30 01:32:25.909377 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:25.909358 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9df52a01-0c8d-44ba-9fd7-9cb2fa41947a-tls-certs\") pod \"distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq\" (UID: \"9df52a01-0c8d-44ba-9fd7-9cb2fa41947a\") " pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" Oct 30 01:32:25.910054 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:25.908779 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9df52a01-0c8d-44ba-9fd7-9cb2fa41947a-model-cache\") pod \"distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq\" (UID: \"9df52a01-0c8d-44ba-9fd7-9cb2fa41947a\") " pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" Oct 30 01:32:25.910659 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:25.908029 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9df52a01-0c8d-44ba-9fd7-9cb2fa41947a-home\") pod \"distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq\" (UID: \"9df52a01-0c8d-44ba-9fd7-9cb2fa41947a\") " pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" Oct 30 01:32:25.910799 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:25.906420 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9df52a01-0c8d-44ba-9fd7-9cb2fa41947a-tmp-dir\") pod \"distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq\" (UID: \"9df52a01-0c8d-44ba-9fd7-9cb2fa41947a\") " pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" Oct 30 01:32:25.921262 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:25.921236 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9df52a01-0c8d-44ba-9fd7-9cb2fa41947a-tls-certs\") pod \"distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq\" (UID: \"9df52a01-0c8d-44ba-9fd7-9cb2fa41947a\") " pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" Oct 30 01:32:25.962669 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:25.962635 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9df52a01-0c8d-44ba-9fd7-9cb2fa41947a-dshm\") pod \"distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq\" (UID: \"9df52a01-0c8d-44ba-9fd7-9cb2fa41947a\") " pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" Oct 30 01:32:25.963744 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:25.963715 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr87s\" (UniqueName: \"kubernetes.io/projected/9df52a01-0c8d-44ba-9fd7-9cb2fa41947a-kube-api-access-dr87s\") pod \"distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq\" (UID: \"9df52a01-0c8d-44ba-9fd7-9cb2fa41947a\") " pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" Oct 30 01:32:26.018302 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:26.018117 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" Oct 30 01:32:26.324933 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:26.324903 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq"] Oct 30 01:32:26.327805 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:32:26.327780 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9df52a01_0c8d_44ba_9fd7_9cb2fa41947a.slice/crio-9102fa466d8ae33cafe2cc83e8c7128dac2f5e83489999fffa264c714146112b WatchSource:0}: Error finding container 9102fa466d8ae33cafe2cc83e8c7128dac2f5e83489999fffa264c714146112b: Status 404 returned error can't find the container with id 9102fa466d8ae33cafe2cc83e8c7128dac2f5e83489999fffa264c714146112b Oct 30 01:32:26.799932 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:26.799892 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" event={"ID":"9df52a01-0c8d-44ba-9fd7-9cb2fa41947a","Type":"ContainerStarted","Data":"bc8cfd1152fae94f9ecd92156cce1aec6f4ec04f8b2f9e8b155575e07647fb03"} Oct 30 01:32:26.799932 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:26.799938 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" event={"ID":"9df52a01-0c8d-44ba-9fd7-9cb2fa41947a","Type":"ContainerStarted","Data":"9102fa466d8ae33cafe2cc83e8c7128dac2f5e83489999fffa264c714146112b"} Oct 30 01:32:26.841837 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:26.841786 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" podStartSLOduration=1.841771609 podStartE2EDuration="1.841771609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:26.839360109 +0000 UTC m=+870.417647445" watchObservedRunningTime="2026-10-30 01:32:26.841771609 +0000 UTC m=+870.420058978" Oct 30 01:32:27.018650 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:27.018602 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" Oct 30 01:32:31.887186 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:31.887152 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5"] Oct 30 01:32:31.891349 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:31.891329 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5" Oct 30 01:32:31.922238 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:31.922061 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5"] Oct 30 01:32:31.945127 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:31.945098 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23246593-d27c-4d51-a771-ae5b61df6fbb-credential-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5\" (UID: \"23246593-d27c-4d51-a771-ae5b61df6fbb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5" Oct 30 01:32:31.945312 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:31.945147 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23246593-d27c-4d51-a771-ae5b61df6fbb-istiod-ca-cert\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5\" (UID: \"23246593-d27c-4d51-a771-ae5b61df6fbb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5" Oct 30 01:32:31.945312 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:31.945210 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23246593-d27c-4d51-a771-ae5b61df6fbb-istio-token\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5\" (UID: \"23246593-d27c-4d51-a771-ae5b61df6fbb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5" Oct 30 01:32:31.945312 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:31.945231 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23246593-d27c-4d51-a771-ae5b61df6fbb-workload-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5\" (UID: \"23246593-d27c-4d51-a771-ae5b61df6fbb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5" Oct 30 01:32:31.945312 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:31.945302 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2wtc\" (UniqueName: \"kubernetes.io/projected/23246593-d27c-4d51-a771-ae5b61df6fbb-kube-api-access-r2wtc\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5\" (UID: \"23246593-d27c-4d51-a771-ae5b61df6fbb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5" Oct 30 01:32:31.945638 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:31.945358 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23246593-d27c-4d51-a771-ae5b61df6fbb-istio-envoy\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5\" (UID: \"23246593-d27c-4d51-a771-ae5b61df6fbb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5" Oct 30 01:32:31.945638 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:31.945471 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23246593-d27c-4d51-a771-ae5b61df6fbb-istio-data\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5\" (UID: \"23246593-d27c-4d51-a771-ae5b61df6fbb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5" Oct 30 01:32:31.945638 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:31.945572 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23246593-d27c-4d51-a771-ae5b61df6fbb-istio-podinfo\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5\" (UID: \"23246593-d27c-4d51-a771-ae5b61df6fbb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5" Oct 30 01:32:31.945807 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:31.945643 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23246593-d27c-4d51-a771-ae5b61df6fbb-workload-certs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5\" (UID: \"23246593-d27c-4d51-a771-ae5b61df6fbb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5" Oct 30 01:32:32.047193 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:32.047127 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23246593-d27c-4d51-a771-ae5b61df6fbb-istio-data\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5\" (UID: \"23246593-d27c-4d51-a771-ae5b61df6fbb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5" Oct 30 01:32:32.047357 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:32.047246 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23246593-d27c-4d51-a771-ae5b61df6fbb-istio-podinfo\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5\" (UID: \"23246593-d27c-4d51-a771-ae5b61df6fbb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5" Oct 30 01:32:32.047357 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:32.047302 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23246593-d27c-4d51-a771-ae5b61df6fbb-workload-certs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5\" (UID: \"23246593-d27c-4d51-a771-ae5b61df6fbb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5" Oct 30 01:32:32.047357 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:32.047345 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23246593-d27c-4d51-a771-ae5b61df6fbb-credential-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5\" (UID: \"23246593-d27c-4d51-a771-ae5b61df6fbb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5" Oct 30 01:32:32.047582 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:32.047406 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23246593-d27c-4d51-a771-ae5b61df6fbb-istiod-ca-cert\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5\" (UID: \"23246593-d27c-4d51-a771-ae5b61df6fbb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5" Oct 30 01:32:32.047582 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:32.047454 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23246593-d27c-4d51-a771-ae5b61df6fbb-istio-token\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5\" (UID: \"23246593-d27c-4d51-a771-ae5b61df6fbb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5" Oct 30 01:32:32.047582 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:32.047485 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23246593-d27c-4d51-a771-ae5b61df6fbb-workload-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5\" (UID: \"23246593-d27c-4d51-a771-ae5b61df6fbb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5" Oct 30 01:32:32.047582 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:32.047522 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2wtc\" (UniqueName: \"kubernetes.io/projected/23246593-d27c-4d51-a771-ae5b61df6fbb-kube-api-access-r2wtc\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5\" (UID: \"23246593-d27c-4d51-a771-ae5b61df6fbb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5" Oct 30 01:32:32.047791 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:32.047587 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23246593-d27c-4d51-a771-ae5b61df6fbb-istio-envoy\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5\" (UID: \"23246593-d27c-4d51-a771-ae5b61df6fbb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5" Oct 30 01:32:32.047791 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:32.047584 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23246593-d27c-4d51-a771-ae5b61df6fbb-istio-data\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5\" (UID: \"23246593-d27c-4d51-a771-ae5b61df6fbb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5" Oct 30 01:32:32.047905 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:32.047857 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23246593-d27c-4d51-a771-ae5b61df6fbb-workload-certs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5\" (UID: \"23246593-d27c-4d51-a771-ae5b61df6fbb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5" Oct 30 01:32:32.048314 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:32.048288 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23246593-d27c-4d51-a771-ae5b61df6fbb-credential-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5\" (UID: \"23246593-d27c-4d51-a771-ae5b61df6fbb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5" Oct 30 01:32:32.048412 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:32.048389 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23246593-d27c-4d51-a771-ae5b61df6fbb-workload-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5\" (UID: \"23246593-d27c-4d51-a771-ae5b61df6fbb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5" Oct 30 01:32:32.048462 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:32.048401 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23246593-d27c-4d51-a771-ae5b61df6fbb-istiod-ca-cert\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5\" (UID: \"23246593-d27c-4d51-a771-ae5b61df6fbb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5" Oct 30 01:32:32.050013 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:32.049988 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23246593-d27c-4d51-a771-ae5b61df6fbb-istio-envoy\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5\" (UID: \"23246593-d27c-4d51-a771-ae5b61df6fbb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5" Oct 30 01:32:32.050101 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:32.050018 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23246593-d27c-4d51-a771-ae5b61df6fbb-istio-podinfo\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5\" (UID: \"23246593-d27c-4d51-a771-ae5b61df6fbb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5" Oct 30 01:32:32.066899 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:32.066873 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23246593-d27c-4d51-a771-ae5b61df6fbb-istio-token\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5\" (UID: \"23246593-d27c-4d51-a771-ae5b61df6fbb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5" Oct 30 01:32:32.067105 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:32.067084 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2wtc\" (UniqueName: \"kubernetes.io/projected/23246593-d27c-4d51-a771-ae5b61df6fbb-kube-api-access-r2wtc\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5\" (UID: \"23246593-d27c-4d51-a771-ae5b61df6fbb\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5" Oct 30 01:32:32.205081 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:32.204998 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5" Oct 30 01:32:32.520521 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:32.520311 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5"] Oct 30 01:32:32.523408 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:32:32.523382 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23246593_d27c_4d51_a771_ae5b61df6fbb.slice/crio-042737ca5999f38f768fb329f501ae50435718bc1f97b296efd80b4fbe3d05f3 WatchSource:0}: Error finding container 042737ca5999f38f768fb329f501ae50435718bc1f97b296efd80b4fbe3d05f3: Status 404 returned error can't find the container with id 042737ca5999f38f768fb329f501ae50435718bc1f97b296efd80b4fbe3d05f3 Oct 30 01:32:32.525872 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:32.525844 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:32:32.525940 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:32.525907 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:32:32.525977 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:32.525944 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:32:32.846081 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:32.846041 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5" event={"ID":"23246593-d27c-4d51-a771-ae5b61df6fbb","Type":"ContainerStarted","Data":"341ea0673c6deb48b8411452f7544c7454366302f9c16bfa3c92d9cccb0fe71d"} Oct 30 01:32:32.846279 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:32.846107 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5" event={"ID":"23246593-d27c-4d51-a771-ae5b61df6fbb","Type":"ContainerStarted","Data":"042737ca5999f38f768fb329f501ae50435718bc1f97b296efd80b4fbe3d05f3"} Oct 30 01:32:32.887601 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:32.887513 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5" podStartSLOduration=1.8874996880000001 podStartE2EDuration="1.887499688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:32.886685327 +0000 UTC m=+876.464972664" watchObservedRunningTime="2026-10-30 01:32:32.887499688 +0000 UTC m=+876.465787023" Oct 30 01:32:33.205672 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:33.205574 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5" Oct 30 01:32:33.207177 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:33.207150 6221 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 01:32:33.207314 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:33.207205 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5" podUID="23246593-d27c-4d51-a771-ae5b61df6fbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 01:32:34.206396 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:34.206361 6221 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 01:32:34.206908 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:34.206431 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5" podUID="23246593-d27c-4d51-a771-ae5b61df6fbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 01:32:35.205799 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:35.205765 6221 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 01:32:35.205993 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:35.205831 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5" podUID="23246593-d27c-4d51-a771-ae5b61df6fbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 01:32:36.050557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:36.046965 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" Oct 30 01:32:36.168616 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:36.168180 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" Oct 30 01:32:36.206312 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:36.206273 6221 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 01:32:36.206487 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:36.206336 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5" podUID="23246593-d27c-4d51-a771-ae5b61df6fbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 01:32:36.881734 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:36.881704 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" Oct 30 01:32:37.206077 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:37.205992 6221 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 01:32:37.206077 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:37.206051 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5" podUID="23246593-d27c-4d51-a771-ae5b61df6fbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 01:32:38.206010 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:38.205975 6221 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 01:32:38.206207 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:38.206036 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5" podUID="23246593-d27c-4d51-a771-ae5b61df6fbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 01:32:39.205471 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:39.205431 6221 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 01:32:39.205674 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:39.205512 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5" podUID="23246593-d27c-4d51-a771-ae5b61df6fbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 01:32:39.831084 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:39.831050 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d"] Oct 30 01:32:39.836918 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:39.836892 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d" Oct 30 01:32:39.877876 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:39.877798 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d"] Oct 30 01:32:40.011228 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.011179 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32d77c5a-d20a-4c87-94a5-bdca98af5135-istio-envoy\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d\" (UID: \"32d77c5a-d20a-4c87-94a5-bdca98af5135\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d" Oct 30 01:32:40.011609 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.011583 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32d77c5a-d20a-4c87-94a5-bdca98af5135-istio-data\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d\" (UID: \"32d77c5a-d20a-4c87-94a5-bdca98af5135\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d" Oct 30 01:32:40.011765 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.011748 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32d77c5a-d20a-4c87-94a5-bdca98af5135-workload-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d\" (UID: \"32d77c5a-d20a-4c87-94a5-bdca98af5135\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d" Oct 30 01:32:40.011876 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.011862 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32d77c5a-d20a-4c87-94a5-bdca98af5135-istio-token\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d\" (UID: \"32d77c5a-d20a-4c87-94a5-bdca98af5135\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d" Oct 30 01:32:40.012024 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.012009 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32d77c5a-d20a-4c87-94a5-bdca98af5135-istio-podinfo\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d\" (UID: \"32d77c5a-d20a-4c87-94a5-bdca98af5135\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d" Oct 30 01:32:40.012152 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.012135 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrrpr\" (UniqueName: \"kubernetes.io/projected/32d77c5a-d20a-4c87-94a5-bdca98af5135-kube-api-access-lrrpr\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d\" (UID: \"32d77c5a-d20a-4c87-94a5-bdca98af5135\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d" Oct 30 01:32:40.012265 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.012251 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32d77c5a-d20a-4c87-94a5-bdca98af5135-istiod-ca-cert\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d\" (UID: \"32d77c5a-d20a-4c87-94a5-bdca98af5135\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d" Oct 30 01:32:40.012422 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.012409 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32d77c5a-d20a-4c87-94a5-bdca98af5135-workload-certs\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d\" (UID: \"32d77c5a-d20a-4c87-94a5-bdca98af5135\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d" Oct 30 01:32:40.012663 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.012581 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32d77c5a-d20a-4c87-94a5-bdca98af5135-credential-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d\" (UID: \"32d77c5a-d20a-4c87-94a5-bdca98af5135\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d" Oct 30 01:32:40.114303 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.114200 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32d77c5a-d20a-4c87-94a5-bdca98af5135-istio-envoy\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d\" (UID: \"32d77c5a-d20a-4c87-94a5-bdca98af5135\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d" Oct 30 01:32:40.114303 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.114293 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32d77c5a-d20a-4c87-94a5-bdca98af5135-istio-data\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d\" (UID: \"32d77c5a-d20a-4c87-94a5-bdca98af5135\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d" Oct 30 01:32:40.114592 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.114336 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32d77c5a-d20a-4c87-94a5-bdca98af5135-workload-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d\" (UID: \"32d77c5a-d20a-4c87-94a5-bdca98af5135\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d" Oct 30 01:32:40.114592 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.114361 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32d77c5a-d20a-4c87-94a5-bdca98af5135-istio-token\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d\" (UID: \"32d77c5a-d20a-4c87-94a5-bdca98af5135\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d" Oct 30 01:32:40.114592 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.114404 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32d77c5a-d20a-4c87-94a5-bdca98af5135-istio-podinfo\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d\" (UID: \"32d77c5a-d20a-4c87-94a5-bdca98af5135\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d" Oct 30 01:32:40.114592 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.114447 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrrpr\" (UniqueName: \"kubernetes.io/projected/32d77c5a-d20a-4c87-94a5-bdca98af5135-kube-api-access-lrrpr\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d\" (UID: \"32d77c5a-d20a-4c87-94a5-bdca98af5135\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d" Oct 30 01:32:40.114592 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.114480 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32d77c5a-d20a-4c87-94a5-bdca98af5135-istiod-ca-cert\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d\" (UID: \"32d77c5a-d20a-4c87-94a5-bdca98af5135\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d" Oct 30 01:32:40.114592 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.114578 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32d77c5a-d20a-4c87-94a5-bdca98af5135-workload-certs\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d\" (UID: \"32d77c5a-d20a-4c87-94a5-bdca98af5135\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d" Oct 30 01:32:40.114909 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.114656 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32d77c5a-d20a-4c87-94a5-bdca98af5135-credential-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d\" (UID: \"32d77c5a-d20a-4c87-94a5-bdca98af5135\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d" Oct 30 01:32:40.115203 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.115177 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32d77c5a-d20a-4c87-94a5-bdca98af5135-credential-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d\" (UID: \"32d77c5a-d20a-4c87-94a5-bdca98af5135\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d" Oct 30 01:32:40.119962 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.115762 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32d77c5a-d20a-4c87-94a5-bdca98af5135-workload-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d\" (UID: \"32d77c5a-d20a-4c87-94a5-bdca98af5135\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d" Oct 30 01:32:40.119962 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.116015 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32d77c5a-d20a-4c87-94a5-bdca98af5135-istio-data\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d\" (UID: \"32d77c5a-d20a-4c87-94a5-bdca98af5135\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d" Oct 30 01:32:40.119962 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.116946 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32d77c5a-d20a-4c87-94a5-bdca98af5135-istiod-ca-cert\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d\" (UID: \"32d77c5a-d20a-4c87-94a5-bdca98af5135\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d" Oct 30 01:32:40.119962 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.117561 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32d77c5a-d20a-4c87-94a5-bdca98af5135-workload-certs\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d\" (UID: \"32d77c5a-d20a-4c87-94a5-bdca98af5135\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d" Oct 30 01:32:40.139829 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.139797 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32d77c5a-d20a-4c87-94a5-bdca98af5135-istio-podinfo\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d\" (UID: \"32d77c5a-d20a-4c87-94a5-bdca98af5135\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d" Oct 30 01:32:40.146020 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.143426 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32d77c5a-d20a-4c87-94a5-bdca98af5135-istio-token\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d\" (UID: \"32d77c5a-d20a-4c87-94a5-bdca98af5135\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d" Oct 30 01:32:40.146020 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.144367 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrrpr\" (UniqueName: \"kubernetes.io/projected/32d77c5a-d20a-4c87-94a5-bdca98af5135-kube-api-access-lrrpr\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d\" (UID: \"32d77c5a-d20a-4c87-94a5-bdca98af5135\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d" Oct 30 01:32:40.146020 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.145058 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32d77c5a-d20a-4c87-94a5-bdca98af5135-istio-envoy\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d\" (UID: \"32d77c5a-d20a-4c87-94a5-bdca98af5135\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d" Oct 30 01:32:40.186402 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.186375 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d" Oct 30 01:32:40.211063 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.211037 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5" Oct 30 01:32:40.211456 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.211432 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5" Oct 30 01:32:40.212317 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.212295 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5" Oct 30 01:32:40.503141 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.502993 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d"] Oct 30 01:32:40.505468 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:32:40.505442 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32d77c5a_d20a_4c87_94a5_bdca98af5135.slice/crio-9c8f86e7c15c56f5789cf6d24b84033cba1c37f6817765370b6a28a724f3619b WatchSource:0}: Error finding container 9c8f86e7c15c56f5789cf6d24b84033cba1c37f6817765370b6a28a724f3619b: Status 404 returned error can't find the container with id 9c8f86e7c15c56f5789cf6d24b84033cba1c37f6817765370b6a28a724f3619b Oct 30 01:32:40.507823 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.507779 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:32:40.507913 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.507866 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:32:40.507913 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.507905 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:32:40.730976 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.730942 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q"] Oct 30 01:32:40.735008 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.734987 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q" Oct 30 01:32:40.764107 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.764027 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q"] Oct 30 01:32:40.896699 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.896660 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d" event={"ID":"32d77c5a-d20a-4c87-94a5-bdca98af5135","Type":"ContainerStarted","Data":"6f1a5ae328f5e5af65f67acb8457b8af4f5930265a59cf780a9af3158a236083"} Oct 30 01:32:40.897126 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.896717 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d" event={"ID":"32d77c5a-d20a-4c87-94a5-bdca98af5135","Type":"ContainerStarted","Data":"9c8f86e7c15c56f5789cf6d24b84033cba1c37f6817765370b6a28a724f3619b"} Oct 30 01:32:40.928743 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.928713 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efc2969c-be45-4d14-af7d-9f8f60e11317-istio-data\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q\" (UID: \"efc2969c-be45-4d14-af7d-9f8f60e11317\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q" Oct 30 01:32:40.928910 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.928832 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efc2969c-be45-4d14-af7d-9f8f60e11317-istiod-ca-cert\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q\" (UID: \"efc2969c-be45-4d14-af7d-9f8f60e11317\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q" Oct 30 01:32:40.928910 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.928890 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efc2969c-be45-4d14-af7d-9f8f60e11317-istio-podinfo\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q\" (UID: \"efc2969c-be45-4d14-af7d-9f8f60e11317\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q" Oct 30 01:32:40.929008 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.928942 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efc2969c-be45-4d14-af7d-9f8f60e11317-workload-certs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q\" (UID: \"efc2969c-be45-4d14-af7d-9f8f60e11317\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q" Oct 30 01:32:40.929101 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.929083 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc2969c-be45-4d14-af7d-9f8f60e11317-credential-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q\" (UID: \"efc2969c-be45-4d14-af7d-9f8f60e11317\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q" Oct 30 01:32:40.929211 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.929114 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efc2969c-be45-4d14-af7d-9f8f60e11317-istio-envoy\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q\" (UID: \"efc2969c-be45-4d14-af7d-9f8f60e11317\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q" Oct 30 01:32:40.929211 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.929145 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efc2969c-be45-4d14-af7d-9f8f60e11317-istio-token\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q\" (UID: \"efc2969c-be45-4d14-af7d-9f8f60e11317\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q" Oct 30 01:32:40.929278 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.929250 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc2969c-be45-4d14-af7d-9f8f60e11317-workload-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q\" (UID: \"efc2969c-be45-4d14-af7d-9f8f60e11317\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q" Oct 30 01:32:40.929312 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.929290 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfhpn\" (UniqueName: \"kubernetes.io/projected/efc2969c-be45-4d14-af7d-9f8f60e11317-kube-api-access-kfhpn\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q\" (UID: \"efc2969c-be45-4d14-af7d-9f8f60e11317\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q" Oct 30 01:32:40.943704 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:40.943659 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d" podStartSLOduration=1.943647297 podStartE2EDuration="1.943647297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:40.942881133 +0000 UTC m=+884.521168482" watchObservedRunningTime="2026-10-30 01:32:40.943647297 +0000 UTC m=+884.521934633" Oct 30 01:32:41.031616 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:41.030331 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efc2969c-be45-4d14-af7d-9f8f60e11317-workload-certs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q\" (UID: \"efc2969c-be45-4d14-af7d-9f8f60e11317\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q" Oct 30 01:32:41.031616 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:41.030423 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc2969c-be45-4d14-af7d-9f8f60e11317-credential-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q\" (UID: \"efc2969c-be45-4d14-af7d-9f8f60e11317\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q" Oct 30 01:32:41.031616 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:41.030448 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efc2969c-be45-4d14-af7d-9f8f60e11317-istio-envoy\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q\" (UID: \"efc2969c-be45-4d14-af7d-9f8f60e11317\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q" Oct 30 01:32:41.031616 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:41.030480 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efc2969c-be45-4d14-af7d-9f8f60e11317-istio-token\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q\" (UID: \"efc2969c-be45-4d14-af7d-9f8f60e11317\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q" Oct 30 01:32:41.031616 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:41.030519 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc2969c-be45-4d14-af7d-9f8f60e11317-workload-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q\" (UID: \"efc2969c-be45-4d14-af7d-9f8f60e11317\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q" Oct 30 01:32:41.031999 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:41.031699 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfhpn\" (UniqueName: \"kubernetes.io/projected/efc2969c-be45-4d14-af7d-9f8f60e11317-kube-api-access-kfhpn\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q\" (UID: \"efc2969c-be45-4d14-af7d-9f8f60e11317\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q" Oct 30 01:32:41.031999 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:41.031823 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efc2969c-be45-4d14-af7d-9f8f60e11317-istio-data\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q\" (UID: \"efc2969c-be45-4d14-af7d-9f8f60e11317\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q" Oct 30 01:32:41.031999 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:41.031966 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efc2969c-be45-4d14-af7d-9f8f60e11317-istiod-ca-cert\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q\" (UID: \"efc2969c-be45-4d14-af7d-9f8f60e11317\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q" Oct 30 01:32:41.032155 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:41.032084 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efc2969c-be45-4d14-af7d-9f8f60e11317-istio-podinfo\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q\" (UID: \"efc2969c-be45-4d14-af7d-9f8f60e11317\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q" Oct 30 01:32:41.032580 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:41.032521 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc2969c-be45-4d14-af7d-9f8f60e11317-workload-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q\" (UID: \"efc2969c-be45-4d14-af7d-9f8f60e11317\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q" Oct 30 01:32:41.032741 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:41.032662 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc2969c-be45-4d14-af7d-9f8f60e11317-credential-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q\" (UID: \"efc2969c-be45-4d14-af7d-9f8f60e11317\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q" Oct 30 01:32:41.032921 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:41.032897 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efc2969c-be45-4d14-af7d-9f8f60e11317-workload-certs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q\" (UID: \"efc2969c-be45-4d14-af7d-9f8f60e11317\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q" Oct 30 01:32:41.033307 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:41.033272 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efc2969c-be45-4d14-af7d-9f8f60e11317-istio-data\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q\" (UID: \"efc2969c-be45-4d14-af7d-9f8f60e11317\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q" Oct 30 01:32:41.033497 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:41.033478 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efc2969c-be45-4d14-af7d-9f8f60e11317-istiod-ca-cert\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q\" (UID: \"efc2969c-be45-4d14-af7d-9f8f60e11317\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q" Oct 30 01:32:41.034658 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:41.034630 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efc2969c-be45-4d14-af7d-9f8f60e11317-istio-envoy\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q\" (UID: \"efc2969c-be45-4d14-af7d-9f8f60e11317\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q" Oct 30 01:32:41.034771 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:41.034753 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efc2969c-be45-4d14-af7d-9f8f60e11317-istio-podinfo\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q\" (UID: \"efc2969c-be45-4d14-af7d-9f8f60e11317\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q" Oct 30 01:32:41.054169 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:41.054141 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efc2969c-be45-4d14-af7d-9f8f60e11317-istio-token\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q\" (UID: \"efc2969c-be45-4d14-af7d-9f8f60e11317\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q" Oct 30 01:32:41.054261 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:41.054240 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfhpn\" (UniqueName: \"kubernetes.io/projected/efc2969c-be45-4d14-af7d-9f8f60e11317-kube-api-access-kfhpn\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q\" (UID: \"efc2969c-be45-4d14-af7d-9f8f60e11317\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q" Oct 30 01:32:41.186505 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:41.186467 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d" Oct 30 01:32:41.188146 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:41.188114 6221 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 01:32:41.188288 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:41.188185 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d" podUID="32d77c5a-d20a-4c87-94a5-bdca98af5135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 01:32:41.350113 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:41.350032 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q" Oct 30 01:32:41.676180 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:41.676072 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-wxkcd"] Oct 30 01:32:41.682433 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:41.682407 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-wxkcd" Oct 30 01:32:41.683086 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:41.683008 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q"] Oct 30 01:32:41.686920 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:32:41.686894 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefc2969c_be45_4d14_af7d_9f8f60e11317.slice/crio-d6c5025c312a62d132b9f7473e08880f4ce075f6e56c1be28d06a75c69a94d6e WatchSource:0}: Error finding container d6c5025c312a62d132b9f7473e08880f4ce075f6e56c1be28d06a75c69a94d6e: Status 404 returned error can't find the container with id d6c5025c312a62d132b9f7473e08880f4ce075f6e56c1be28d06a75c69a94d6e Oct 30 01:32:41.689573 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:41.689546 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:32:41.689640 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:41.689605 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:32:41.689640 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:41.689634 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:32:41.699451 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:41.697417 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-wxkcd"] Oct 30 01:32:41.743637 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:41.743610 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-278fe7d5-dockercfg-jlrw2\"" Oct 30 01:32:41.750047 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:41.750016 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41211dad-aa70-47ff-b6bb-12426581ab00-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-wxkcd\" (UID: \"41211dad-aa70-47ff-b6bb-12426581ab00\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-wxkcd" Oct 30 01:32:41.750348 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:41.750326 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c5xx\" (UniqueName: \"kubernetes.io/projected/41211dad-aa70-47ff-b6bb-12426581ab00-kube-api-access-8c5xx\") pod \"payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-wxkcd\" (UID: \"41211dad-aa70-47ff-b6bb-12426581ab00\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-wxkcd" Oct 30 01:32:41.767596 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:41.767570 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-278fe7d5\"" Oct 30 01:32:41.851596 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:41.851523 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41211dad-aa70-47ff-b6bb-12426581ab00-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-wxkcd\" (UID: \"41211dad-aa70-47ff-b6bb-12426581ab00\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-wxkcd" Oct 30 01:32:41.851796 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:41.851650 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c5xx\" (UniqueName: \"kubernetes.io/projected/41211dad-aa70-47ff-b6bb-12426581ab00-kube-api-access-8c5xx\") pod \"payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-wxkcd\" (UID: \"41211dad-aa70-47ff-b6bb-12426581ab00\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-wxkcd" Oct 30 01:32:41.852708 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:41.852672 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41211dad-aa70-47ff-b6bb-12426581ab00-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-wxkcd\" (UID: \"41211dad-aa70-47ff-b6bb-12426581ab00\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-wxkcd" Oct 30 01:32:41.872342 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:41.872240 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c5xx\" (UniqueName: \"kubernetes.io/projected/41211dad-aa70-47ff-b6bb-12426581ab00-kube-api-access-8c5xx\") pod \"payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-wxkcd\" (UID: \"41211dad-aa70-47ff-b6bb-12426581ab00\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-wxkcd" Oct 30 01:32:41.998614 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:41.997051 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q" event={"ID":"efc2969c-be45-4d14-af7d-9f8f60e11317","Type":"ContainerStarted","Data":"cc5984444ae87516d39d551f30693269103f569ec72696325de23b17f3957fed"} Oct 30 01:32:41.998614 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:41.997113 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q" event={"ID":"efc2969c-be45-4d14-af7d-9f8f60e11317","Type":"ContainerStarted","Data":"d6c5025c312a62d132b9f7473e08880f4ce075f6e56c1be28d06a75c69a94d6e"} Oct 30 01:32:42.067558 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:42.053279 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-wxkcd" Oct 30 01:32:42.110063 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:42.109978 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q" podStartSLOduration=2.109894165 podStartE2EDuration="2.109894165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:42.047902184 +0000 UTC m=+885.626189565" watchObservedRunningTime="2026-10-30 01:32:42.109894165 +0000 UTC m=+885.688181501" Oct 30 01:32:42.188235 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:42.188201 6221 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 01:32:42.188412 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:42.188253 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d" podUID="32d77c5a-d20a-4c87-94a5-bdca98af5135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 01:32:42.350912 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:42.350879 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q" Oct 30 01:32:42.352228 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:42.352195 6221 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 01:32:42.352388 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:42.352250 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q" podUID="efc2969c-be45-4d14-af7d-9f8f60e11317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 01:32:42.451488 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:42.451462 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-wxkcd"] Oct 30 01:32:42.453823 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:32:42.453796 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41211dad_aa70_47ff_b6bb_12426581ab00.slice/crio-279351eb413defaeab39a411443fe4e6f8045d218e588660d12c26dec30e5045 WatchSource:0}: Error finding container 279351eb413defaeab39a411443fe4e6f8045d218e588660d12c26dec30e5045: Status 404 returned error can't find the container with id 279351eb413defaeab39a411443fe4e6f8045d218e588660d12c26dec30e5045 Oct 30 01:32:43.001939 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:43.001900 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-wxkcd" event={"ID":"41211dad-aa70-47ff-b6bb-12426581ab00","Type":"ContainerStarted","Data":"465226a307196f73a364d6d233cfd00a766f70de1bcf8c822661103ef0647471"} Oct 30 01:32:43.002339 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:43.001964 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-wxkcd" event={"ID":"41211dad-aa70-47ff-b6bb-12426581ab00","Type":"ContainerStarted","Data":"279351eb413defaeab39a411443fe4e6f8045d218e588660d12c26dec30e5045"} Oct 30 01:32:43.002339 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:43.002218 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-wxkcd" Oct 30 01:32:43.187110 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:43.187059 6221 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 01:32:43.187399 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:43.187351 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d" podUID="32d77c5a-d20a-4c87-94a5-bdca98af5135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 01:32:43.356557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:43.353955 6221 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 01:32:43.356557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:43.354016 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q" podUID="efc2969c-be45-4d14-af7d-9f8f60e11317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 01:32:44.186883 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:44.186852 6221 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 01:32:44.187291 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:44.186902 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d" podUID="32d77c5a-d20a-4c87-94a5-bdca98af5135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 01:32:44.354556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:44.353979 6221 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 01:32:44.354556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:44.354050 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q" podUID="efc2969c-be45-4d14-af7d-9f8f60e11317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 01:32:45.187218 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:45.187177 6221 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 01:32:45.187670 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:45.187233 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d" podUID="32d77c5a-d20a-4c87-94a5-bdca98af5135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 01:32:45.356059 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:45.356019 6221 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 01:32:45.356342 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:45.356308 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q" podUID="efc2969c-be45-4d14-af7d-9f8f60e11317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 01:32:46.186946 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:46.186910 6221 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 01:32:46.187132 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:46.186964 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d" podUID="32d77c5a-d20a-4c87-94a5-bdca98af5135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 01:32:46.355584 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:46.353932 6221 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 01:32:46.355584 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:46.353994 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q" podUID="efc2969c-be45-4d14-af7d-9f8f60e11317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 01:32:47.187086 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:47.187049 6221 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 01:32:47.187280 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:47.187116 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d" podUID="32d77c5a-d20a-4c87-94a5-bdca98af5135" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 01:32:47.351000 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:47.350962 6221 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 01:32:47.351214 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:47.351027 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q" podUID="efc2969c-be45-4d14-af7d-9f8f60e11317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 01:32:48.255064 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:48.255031 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d" Oct 30 01:32:48.255641 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:48.255517 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d" Oct 30 01:32:48.257088 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:48.257068 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d" Oct 30 01:32:48.300421 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:48.300364 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-wxkcd" podStartSLOduration=7.300351105 podStartE2EDuration="7.300351105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:43.060870685 +0000 UTC m=+886.639158021" watchObservedRunningTime="2026-10-30 01:32:48.300351105 +0000 UTC m=+891.878638441" Oct 30 01:32:48.350507 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:48.350474 6221 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 01:32:48.350687 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:48.350558 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q" podUID="efc2969c-be45-4d14-af7d-9f8f60e11317" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 01:32:48.692813 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:48.692779 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:32:49.376576 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:49.373918 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q" Oct 30 01:32:49.376576 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:49.375123 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q" Oct 30 01:32:49.386089 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:49.386051 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q" Oct 30 01:32:51.611556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:51.552116 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" podUID="ea9583aa-8259-4c9b-932e-2d6559008d86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:32:51.630186 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:51.625622 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk" podUID="f4ae494e-81ec-4039-ae07-7ed1bf345f0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Oct 30 01:32:51.630186 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:51.626347 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" podUID="018c3fe2-3e67-45ed-a3d7-d70f3fe5b956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:32:51.722556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:51.721627 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" podUID="3b3c0e08-1371-4184-816a-d8b501526190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:32:51.929646 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:51.876654 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" podUID="9df52a01-0c8d-44ba-9fd7-9cb2fa41947a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:32:52.106672 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:32:52.106629 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:32:52.122272 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:52.122223 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" podUID="354ec282-b972-4fca-b4e3-1e33ef461f2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:32:52.628558 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:52.625167 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" podUID="ea9583aa-8259-4c9b-932e-2d6559008d86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:32:52.628558 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:52.625811 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk" podUID="f4ae494e-81ec-4039-ae07-7ed1bf345f0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:32:52.628558 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:52.626102 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" Oct 30 01:32:52.628558 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:52.626161 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk" Oct 30 01:32:52.640804 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:52.640765 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" podUID="018c3fe2-3e67-45ed-a3d7-d70f3fe5b956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 01:32:52.653667 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:52.641560 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" Oct 30 01:32:52.760355 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:52.760307 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" podUID="3b3c0e08-1371-4184-816a-d8b501526190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:32:52.767601 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:52.767571 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" Oct 30 01:32:53.017636 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:52.888669 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" podUID="9df52a01-0c8d-44ba-9fd7-9cb2fa41947a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:32:53.017636 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:52.958290 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" Oct 30 01:32:53.057622 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:53.057294 6221 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" podUID="354ec282-b972-4fca-b4e3-1e33ef461f2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:32:53.130524 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:53.130479 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" podUID="354ec282-b972-4fca-b4e3-1e33ef461f2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:32:53.131469 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:53.131444 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" Oct 30 01:32:53.133629 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:53.133545 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" Oct 30 01:32:53.256673 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:53.256637 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk" Oct 30 01:32:53.526881 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:53.526849 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" Oct 30 01:32:53.584525 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:53.584367 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" Oct 30 01:32:53.625657 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:53.625617 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" podUID="ea9583aa-8259-4c9b-932e-2d6559008d86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 01:32:53.701553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:53.690779 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" Oct 30 01:32:53.723742 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:53.723714 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" Oct 30 01:32:54.011716 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:54.011689 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-wxkcd" Oct 30 01:32:55.054448 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:55.054413 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-x4w45"] Oct 30 01:32:55.059610 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:55.059588 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-x4w45" Oct 30 01:32:55.076389 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:55.076364 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-x4w45"] Oct 30 01:32:55.085603 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:55.085575 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-278fe7d5-serving-cert\"" Oct 30 01:32:55.086314 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:55.086280 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5371c07a-6563-4db1-9483-32e9300a0581-openshift-service-ca\") pod \"maas-api-e2e-mt-278fe7d5-59f4cbfd44-x4w45\" (UID: \"5371c07a-6563-4db1-9483-32e9300a0581\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-x4w45" Oct 30 01:32:55.086422 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:55.086385 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5371c07a-6563-4db1-9483-32e9300a0581-maas-api-tls\") pod \"maas-api-e2e-mt-278fe7d5-59f4cbfd44-x4w45\" (UID: \"5371c07a-6563-4db1-9483-32e9300a0581\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-x4w45" Oct 30 01:32:55.086481 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:55.086435 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wb9p\" (UniqueName: \"kubernetes.io/projected/5371c07a-6563-4db1-9483-32e9300a0581-kube-api-access-2wb9p\") pod \"maas-api-e2e-mt-278fe7d5-59f4cbfd44-x4w45\" (UID: \"5371c07a-6563-4db1-9483-32e9300a0581\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-x4w45" Oct 30 01:32:55.086571 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:55.086521 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5371c07a-6563-4db1-9483-32e9300a0581-metrics-certs\") pod \"maas-api-e2e-mt-278fe7d5-59f4cbfd44-x4w45\" (UID: \"5371c07a-6563-4db1-9483-32e9300a0581\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-x4w45" Oct 30 01:32:55.188038 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:55.188004 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wb9p\" (UniqueName: \"kubernetes.io/projected/5371c07a-6563-4db1-9483-32e9300a0581-kube-api-access-2wb9p\") pod \"maas-api-e2e-mt-278fe7d5-59f4cbfd44-x4w45\" (UID: \"5371c07a-6563-4db1-9483-32e9300a0581\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-x4w45" Oct 30 01:32:55.188210 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:55.188109 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5371c07a-6563-4db1-9483-32e9300a0581-metrics-certs\") pod \"maas-api-e2e-mt-278fe7d5-59f4cbfd44-x4w45\" (UID: \"5371c07a-6563-4db1-9483-32e9300a0581\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-x4w45" Oct 30 01:32:55.188332 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:55.188226 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5371c07a-6563-4db1-9483-32e9300a0581-openshift-service-ca\") pod \"maas-api-e2e-mt-278fe7d5-59f4cbfd44-x4w45\" (UID: \"5371c07a-6563-4db1-9483-32e9300a0581\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-x4w45" Oct 30 01:32:55.188401 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:55.188328 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5371c07a-6563-4db1-9483-32e9300a0581-maas-api-tls\") pod \"maas-api-e2e-mt-278fe7d5-59f4cbfd44-x4w45\" (UID: \"5371c07a-6563-4db1-9483-32e9300a0581\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-x4w45" Oct 30 01:32:55.188493 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:32:55.188476 6221 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-serving-cert: secret "maas-api-e2e-mt-278fe7d5-serving-cert" not found Oct 30 01:32:55.188695 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:32:55.188679 6221 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5371c07a-6563-4db1-9483-32e9300a0581-maas-api-tls podName:5371c07a-6563-4db1-9483-32e9300a0581 nodeName:}" failed. No retries permitted until 2026-10-30 01:32:55.688661692 +0000 UTC m=+899.266949021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5371c07a-6563-4db1-9483-32e9300a0581-maas-api-tls") pod "maas-api-e2e-mt-278fe7d5-59f4cbfd44-x4w45" (UID: "5371c07a-6563-4db1-9483-32e9300a0581") : secret "maas-api-e2e-mt-278fe7d5-serving-cert" not found Oct 30 01:32:55.188996 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:55.188976 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5371c07a-6563-4db1-9483-32e9300a0581-openshift-service-ca\") pod \"maas-api-e2e-mt-278fe7d5-59f4cbfd44-x4w45\" (UID: \"5371c07a-6563-4db1-9483-32e9300a0581\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-x4w45" Oct 30 01:32:55.190544 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:55.190515 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5371c07a-6563-4db1-9483-32e9300a0581-metrics-certs\") pod \"maas-api-e2e-mt-278fe7d5-59f4cbfd44-x4w45\" (UID: \"5371c07a-6563-4db1-9483-32e9300a0581\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-x4w45" Oct 30 01:32:55.207930 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:55.207902 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wb9p\" (UniqueName: \"kubernetes.io/projected/5371c07a-6563-4db1-9483-32e9300a0581-kube-api-access-2wb9p\") pod \"maas-api-e2e-mt-278fe7d5-59f4cbfd44-x4w45\" (UID: \"5371c07a-6563-4db1-9483-32e9300a0581\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-x4w45" Oct 30 01:32:55.713152 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:55.713121 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5371c07a-6563-4db1-9483-32e9300a0581-maas-api-tls\") pod \"maas-api-e2e-mt-278fe7d5-59f4cbfd44-x4w45\" (UID: \"5371c07a-6563-4db1-9483-32e9300a0581\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-x4w45" Oct 30 01:32:55.715577 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:55.715547 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5371c07a-6563-4db1-9483-32e9300a0581-maas-api-tls\") pod \"maas-api-e2e-mt-278fe7d5-59f4cbfd44-x4w45\" (UID: \"5371c07a-6563-4db1-9483-32e9300a0581\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-x4w45" Oct 30 01:32:55.975796 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:55.975699 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-x4w45" Oct 30 01:32:56.309515 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:56.309490 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-x4w45"] Oct 30 01:32:56.316697 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:56.316041 6221 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:32:56.584368 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:56.584280 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-x4w45" event={"ID":"5371c07a-6563-4db1-9483-32e9300a0581","Type":"ContainerStarted","Data":"4473e51f9457561c7050252b2f025adabfd69ec94163137e508ae836e4b3de3a"} Oct 30 01:32:56.639324 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:56.639293 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2vsns_d332bbb4-c699-44e8-96ba-516d8ca03e95/ovn-acl-logging/0.log" Oct 30 01:32:56.647257 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:56.647230 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2vsns_d332bbb4-c699-44e8-96ba-516d8ca03e95/ovn-acl-logging/0.log" Oct 30 01:32:57.396358 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:57.396314 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr"] Oct 30 01:32:57.400943 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:57.400915 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr" Oct 30 01:32:57.448920 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:57.440046 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr"] Oct 30 01:32:57.486205 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:57.486166 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40e8c5af-0173-4388-b938-ddc9b82ea112-istio-token\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-flsxr\" (UID: \"40e8c5af-0173-4388-b938-ddc9b82ea112\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr" Oct 30 01:32:57.486384 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:57.486221 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e8c5af-0173-4388-b938-ddc9b82ea112-workload-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-flsxr\" (UID: \"40e8c5af-0173-4388-b938-ddc9b82ea112\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr" Oct 30 01:32:57.486384 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:57.486367 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e8c5af-0173-4388-b938-ddc9b82ea112-credential-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-flsxr\" (UID: \"40e8c5af-0173-4388-b938-ddc9b82ea112\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr" Oct 30 01:32:57.486478 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:57.486413 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e8c5af-0173-4388-b938-ddc9b82ea112-istio-envoy\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-flsxr\" (UID: \"40e8c5af-0173-4388-b938-ddc9b82ea112\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr" Oct 30 01:32:57.486615 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:57.486589 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e8c5af-0173-4388-b938-ddc9b82ea112-istiod-ca-cert\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-flsxr\" (UID: \"40e8c5af-0173-4388-b938-ddc9b82ea112\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr" Oct 30 01:32:57.486688 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:57.486629 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm97h\" (UniqueName: \"kubernetes.io/projected/40e8c5af-0173-4388-b938-ddc9b82ea112-kube-api-access-jm97h\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-flsxr\" (UID: \"40e8c5af-0173-4388-b938-ddc9b82ea112\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr" Oct 30 01:32:57.486748 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:57.486725 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e8c5af-0173-4388-b938-ddc9b82ea112-istio-data\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-flsxr\" (UID: \"40e8c5af-0173-4388-b938-ddc9b82ea112\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr" Oct 30 01:32:57.486789 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:57.486774 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e8c5af-0173-4388-b938-ddc9b82ea112-workload-certs\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-flsxr\" (UID: \"40e8c5af-0173-4388-b938-ddc9b82ea112\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr" Oct 30 01:32:57.486821 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:57.486798 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e8c5af-0173-4388-b938-ddc9b82ea112-istio-podinfo\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-flsxr\" (UID: \"40e8c5af-0173-4388-b938-ddc9b82ea112\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr" Oct 30 01:32:57.588015 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:57.587979 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e8c5af-0173-4388-b938-ddc9b82ea112-istiod-ca-cert\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-flsxr\" (UID: \"40e8c5af-0173-4388-b938-ddc9b82ea112\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr" Oct 30 01:32:57.588015 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:57.588014 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm97h\" (UniqueName: \"kubernetes.io/projected/40e8c5af-0173-4388-b938-ddc9b82ea112-kube-api-access-jm97h\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-flsxr\" (UID: \"40e8c5af-0173-4388-b938-ddc9b82ea112\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr" Oct 30 01:32:57.588261 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:57.588047 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e8c5af-0173-4388-b938-ddc9b82ea112-istio-data\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-flsxr\" (UID: \"40e8c5af-0173-4388-b938-ddc9b82ea112\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr" Oct 30 01:32:57.588261 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:57.588078 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e8c5af-0173-4388-b938-ddc9b82ea112-workload-certs\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-flsxr\" (UID: \"40e8c5af-0173-4388-b938-ddc9b82ea112\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr" Oct 30 01:32:57.588261 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:57.588100 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e8c5af-0173-4388-b938-ddc9b82ea112-istio-podinfo\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-flsxr\" (UID: \"40e8c5af-0173-4388-b938-ddc9b82ea112\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr" Oct 30 01:32:57.588261 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:57.588117 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40e8c5af-0173-4388-b938-ddc9b82ea112-istio-token\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-flsxr\" (UID: \"40e8c5af-0173-4388-b938-ddc9b82ea112\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr" Oct 30 01:32:57.588459 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:57.588307 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e8c5af-0173-4388-b938-ddc9b82ea112-workload-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-flsxr\" (UID: \"40e8c5af-0173-4388-b938-ddc9b82ea112\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr" Oct 30 01:32:57.588525 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:57.588475 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e8c5af-0173-4388-b938-ddc9b82ea112-credential-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-flsxr\" (UID: \"40e8c5af-0173-4388-b938-ddc9b82ea112\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr" Oct 30 01:32:57.588605 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:57.588564 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e8c5af-0173-4388-b938-ddc9b82ea112-istio-envoy\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-flsxr\" (UID: \"40e8c5af-0173-4388-b938-ddc9b82ea112\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr" Oct 30 01:32:57.588699 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:57.588648 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e8c5af-0173-4388-b938-ddc9b82ea112-istio-data\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-flsxr\" (UID: \"40e8c5af-0173-4388-b938-ddc9b82ea112\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr" Oct 30 01:32:57.588893 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:57.588816 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e8c5af-0173-4388-b938-ddc9b82ea112-workload-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-flsxr\" (UID: \"40e8c5af-0173-4388-b938-ddc9b82ea112\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr" Oct 30 01:32:57.588980 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:57.588913 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e8c5af-0173-4388-b938-ddc9b82ea112-istiod-ca-cert\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-flsxr\" (UID: \"40e8c5af-0173-4388-b938-ddc9b82ea112\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr" Oct 30 01:32:57.588980 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:57.588957 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e8c5af-0173-4388-b938-ddc9b82ea112-workload-certs\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-flsxr\" (UID: \"40e8c5af-0173-4388-b938-ddc9b82ea112\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr" Oct 30 01:32:57.589081 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:57.589066 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e8c5af-0173-4388-b938-ddc9b82ea112-credential-socket\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-flsxr\" (UID: \"40e8c5af-0173-4388-b938-ddc9b82ea112\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr" Oct 30 01:32:57.591320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:57.591294 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e8c5af-0173-4388-b938-ddc9b82ea112-istio-envoy\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-flsxr\" (UID: \"40e8c5af-0173-4388-b938-ddc9b82ea112\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr" Oct 30 01:32:57.591791 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:57.591769 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e8c5af-0173-4388-b938-ddc9b82ea112-istio-podinfo\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-flsxr\" (UID: \"40e8c5af-0173-4388-b938-ddc9b82ea112\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr" Oct 30 01:32:57.592126 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:57.592105 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-x4w45" event={"ID":"5371c07a-6563-4db1-9483-32e9300a0581","Type":"ContainerStarted","Data":"7a749af26f034e0988a412a349f35edba89574946bb69cfe7380931e74f63330"} Oct 30 01:32:57.592292 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:57.592276 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-x4w45" Oct 30 01:32:57.607259 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:57.607233 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40e8c5af-0173-4388-b938-ddc9b82ea112-istio-token\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-flsxr\" (UID: \"40e8c5af-0173-4388-b938-ddc9b82ea112\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr" Oct 30 01:32:57.607469 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:57.607451 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm97h\" (UniqueName: \"kubernetes.io/projected/40e8c5af-0173-4388-b938-ddc9b82ea112-kube-api-access-jm97h\") pod \"e2e-worker-w3-558327-openshift-default-576847689f-flsxr\" (UID: \"40e8c5af-0173-4388-b938-ddc9b82ea112\") " pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr" Oct 30 01:32:57.633807 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:57.633754 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-x4w45" podStartSLOduration=2.20151265 podStartE2EDuration="2.633739903s" totalImagesPullingTime="432.227253ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:57.63139304 +0000 UTC m=+901.209680376" watchObservedRunningTime="2026-10-30 01:32:57.633739903 +0000 UTC m=+901.212027291" Oct 30 01:32:57.716351 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:57.716268 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr" Oct 30 01:32:58.116361 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:58.116326 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr"] Oct 30 01:32:58.120097 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:32:58.120068 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e8c5af_0173_4388_b938_ddc9b82ea112.slice/crio-c0f4bd7191be185a2dc6ec43f6f5821ed920388c518618c942ed3076eb95abca WatchSource:0}: Error finding container c0f4bd7191be185a2dc6ec43f6f5821ed920388c518618c942ed3076eb95abca: Status 404 returned error can't find the container with id c0f4bd7191be185a2dc6ec43f6f5821ed920388c518618c942ed3076eb95abca Oct 30 01:32:58.122595 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:58.122563 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:32:58.122657 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:58.122634 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:32:58.122700 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:58.122661 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:32:58.600663 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:58.600622 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr" event={"ID":"40e8c5af-0173-4388-b938-ddc9b82ea112","Type":"ContainerStarted","Data":"ebef54a515eebaf98f3f0719c4d2c943c8b073037d929cad5d3f542928cc3c48"} Oct 30 01:32:58.601134 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:58.600677 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr" event={"ID":"40e8c5af-0173-4388-b938-ddc9b82ea112","Type":"ContainerStarted","Data":"c0f4bd7191be185a2dc6ec43f6f5821ed920388c518618c942ed3076eb95abca"} Oct 30 01:32:58.671283 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:58.671232 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr" podStartSLOduration=1.6712170990000002 podStartE2EDuration="1.671217099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:32:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:32:58.669619012 +0000 UTC m=+902.247906349" watchObservedRunningTime="2026-10-30 01:32:58.671217099 +0000 UTC m=+902.249504435" Oct 30 01:32:58.722557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:58.717621 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr" Oct 30 01:32:59.729556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:59.721154 6221 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-flsxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 01:32:59.729556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:32:59.721220 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr" podUID="40e8c5af-0173-4388-b938-ddc9b82ea112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 01:33:00.730557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:00.727247 6221 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-flsxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 01:33:00.730557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:00.727314 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr" podUID="40e8c5af-0173-4388-b938-ddc9b82ea112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 01:33:01.717551 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:01.716676 6221 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-flsxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 01:33:01.717551 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:01.716755 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr" podUID="40e8c5af-0173-4388-b938-ddc9b82ea112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 01:33:02.731788 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:02.731743 6221 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-flsxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 01:33:02.732345 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:02.731807 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr" podUID="40e8c5af-0173-4388-b938-ddc9b82ea112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 01:33:03.631946 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:03.631902 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-x4w45" Oct 30 01:33:03.735765 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:03.735729 6221 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-flsxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 01:33:03.737002 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:03.736963 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr" podUID="40e8c5af-0173-4388-b938-ddc9b82ea112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 01:33:04.716868 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:04.716830 6221 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-flsxr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 01:33:04.717059 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:04.716898 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr" podUID="40e8c5af-0173-4388-b938-ddc9b82ea112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 01:33:05.721817 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:05.721784 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr" Oct 30 01:33:05.722236 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:05.722091 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr" Oct 30 01:33:05.722828 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:05.722804 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr" Oct 30 01:33:13.530892 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:13.530630 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" podUID="018c3fe2-3e67-45ed-a3d7-d70f3fe5b956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:13.620840 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:13.620758 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" podUID="9df52a01-0c8d-44ba-9fd7-9cb2fa41947a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 01:33:13.697674 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:13.697624 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" podUID="ea9583aa-8259-4c9b-932e-2d6559008d86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:13.736671 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:13.736621 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" podUID="354ec282-b972-4fca-b4e3-1e33ef461f2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 01:33:14.152089 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:14.150211 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" podUID="3b3c0e08-1371-4184-816a-d8b501526190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 01:33:14.283588 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:14.282621 6221 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" podUID="ea9583aa-8259-4c9b-932e-2d6559008d86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:14.624839 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:14.623638 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" podUID="9df52a01-0c8d-44ba-9fd7-9cb2fa41947a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 01:33:14.732557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:14.704816 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" podUID="ea9583aa-8259-4c9b-932e-2d6559008d86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:14.766870 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:14.757691 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" podUID="018c3fe2-3e67-45ed-a3d7-d70f3fe5b956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 01:33:14.811225 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:14.811193 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" Oct 30 01:33:14.811553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:14.811515 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" Oct 30 01:33:14.812972 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:14.812937 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" Oct 30 01:33:14.884705 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:14.881762 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" Oct 30 01:33:15.101554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:15.100088 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" Oct 30 01:33:15.168595 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:15.168474 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" podUID="3b3c0e08-1371-4184-816a-d8b501526190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:15.196362 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:15.196331 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" Oct 30 01:33:15.611431 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:15.611397 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" Oct 30 01:33:15.789449 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:15.739991 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" podUID="354ec282-b972-4fca-b4e3-1e33ef461f2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:15.915404 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:15.915318 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" Oct 30 01:33:21.493966 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:33:21.493926 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:33:24.933499 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:24.933462 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp"] Oct 30 01:33:24.938027 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:24.937999 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp" Oct 30 01:33:24.962250 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:24.962222 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp"] Oct 30 01:33:25.006285 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:25.006248 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9pxk\" (UniqueName: \"kubernetes.io/projected/410ab4b3-ac20-4633-8311-763564d670e4-kube-api-access-l9pxk\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp\" (UID: \"410ab4b3-ac20-4633-8311-763564d670e4\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp" Oct 30 01:33:25.006485 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:25.006438 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/410ab4b3-ac20-4633-8311-763564d670e4-workload-certs\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp\" (UID: \"410ab4b3-ac20-4633-8311-763564d670e4\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp" Oct 30 01:33:25.006585 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:25.006483 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/410ab4b3-ac20-4633-8311-763564d670e4-istio-token\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp\" (UID: \"410ab4b3-ac20-4633-8311-763564d670e4\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp" Oct 30 01:33:25.006667 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:25.006629 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/410ab4b3-ac20-4633-8311-763564d670e4-istio-podinfo\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp\" (UID: \"410ab4b3-ac20-4633-8311-763564d670e4\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp" Oct 30 01:33:25.006730 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:25.006716 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/410ab4b3-ac20-4633-8311-763564d670e4-istiod-ca-cert\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp\" (UID: \"410ab4b3-ac20-4633-8311-763564d670e4\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp" Oct 30 01:33:25.006799 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:25.006783 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/410ab4b3-ac20-4633-8311-763564d670e4-credential-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp\" (UID: \"410ab4b3-ac20-4633-8311-763564d670e4\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp" Oct 30 01:33:25.006859 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:25.006817 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/410ab4b3-ac20-4633-8311-763564d670e4-workload-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp\" (UID: \"410ab4b3-ac20-4633-8311-763564d670e4\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp" Oct 30 01:33:25.006859 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:25.006832 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/410ab4b3-ac20-4633-8311-763564d670e4-istio-envoy\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp\" (UID: \"410ab4b3-ac20-4633-8311-763564d670e4\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp" Oct 30 01:33:25.006973 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:25.006870 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/410ab4b3-ac20-4633-8311-763564d670e4-istio-data\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp\" (UID: \"410ab4b3-ac20-4633-8311-763564d670e4\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp" Oct 30 01:33:25.107818 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:25.107779 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/410ab4b3-ac20-4633-8311-763564d670e4-workload-certs\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp\" (UID: \"410ab4b3-ac20-4633-8311-763564d670e4\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp" Oct 30 01:33:25.107818 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:25.107828 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/410ab4b3-ac20-4633-8311-763564d670e4-istio-token\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp\" (UID: \"410ab4b3-ac20-4633-8311-763564d670e4\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp" Oct 30 01:33:25.108060 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:25.107891 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/410ab4b3-ac20-4633-8311-763564d670e4-istio-podinfo\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp\" (UID: \"410ab4b3-ac20-4633-8311-763564d670e4\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp" Oct 30 01:33:25.108060 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:25.107969 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/410ab4b3-ac20-4633-8311-763564d670e4-istiod-ca-cert\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp\" (UID: \"410ab4b3-ac20-4633-8311-763564d670e4\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp" Oct 30 01:33:25.108060 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:25.108025 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/410ab4b3-ac20-4633-8311-763564d670e4-credential-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp\" (UID: \"410ab4b3-ac20-4633-8311-763564d670e4\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp" Oct 30 01:33:25.108220 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:25.108066 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/410ab4b3-ac20-4633-8311-763564d670e4-workload-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp\" (UID: \"410ab4b3-ac20-4633-8311-763564d670e4\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp" Oct 30 01:33:25.108220 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:25.108088 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/410ab4b3-ac20-4633-8311-763564d670e4-istio-envoy\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp\" (UID: \"410ab4b3-ac20-4633-8311-763564d670e4\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp" Oct 30 01:33:25.108220 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:25.108115 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/410ab4b3-ac20-4633-8311-763564d670e4-istio-data\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp\" (UID: \"410ab4b3-ac20-4633-8311-763564d670e4\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp" Oct 30 01:33:25.108220 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:25.108159 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/410ab4b3-ac20-4633-8311-763564d670e4-workload-certs\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp\" (UID: \"410ab4b3-ac20-4633-8311-763564d670e4\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp" Oct 30 01:33:25.108220 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:25.108173 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9pxk\" (UniqueName: \"kubernetes.io/projected/410ab4b3-ac20-4633-8311-763564d670e4-kube-api-access-l9pxk\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp\" (UID: \"410ab4b3-ac20-4633-8311-763564d670e4\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp" Oct 30 01:33:25.108482 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:25.108372 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/410ab4b3-ac20-4633-8311-763564d670e4-credential-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp\" (UID: \"410ab4b3-ac20-4633-8311-763564d670e4\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp" Oct 30 01:33:25.108572 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:25.108511 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/410ab4b3-ac20-4633-8311-763564d670e4-workload-socket\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp\" (UID: \"410ab4b3-ac20-4633-8311-763564d670e4\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp" Oct 30 01:33:25.108765 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:25.108739 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/410ab4b3-ac20-4633-8311-763564d670e4-istio-data\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp\" (UID: \"410ab4b3-ac20-4633-8311-763564d670e4\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp" Oct 30 01:33:25.108873 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:25.108773 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/410ab4b3-ac20-4633-8311-763564d670e4-istiod-ca-cert\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp\" (UID: \"410ab4b3-ac20-4633-8311-763564d670e4\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp" Oct 30 01:33:25.110394 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:25.110367 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/410ab4b3-ac20-4633-8311-763564d670e4-istio-envoy\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp\" (UID: \"410ab4b3-ac20-4633-8311-763564d670e4\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp" Oct 30 01:33:25.110766 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:25.110749 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/410ab4b3-ac20-4633-8311-763564d670e4-istio-podinfo\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp\" (UID: \"410ab4b3-ac20-4633-8311-763564d670e4\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp" Oct 30 01:33:25.127349 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:25.127320 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/410ab4b3-ac20-4633-8311-763564d670e4-istio-token\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp\" (UID: \"410ab4b3-ac20-4633-8311-763564d670e4\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp" Oct 30 01:33:25.127448 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:25.127328 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9pxk\" (UniqueName: \"kubernetes.io/projected/410ab4b3-ac20-4633-8311-763564d670e4-kube-api-access-l9pxk\") pod \"e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp\" (UID: \"410ab4b3-ac20-4633-8311-763564d670e4\") " pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp" Oct 30 01:33:25.252295 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:25.252216 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp" Oct 30 01:33:25.576943 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:25.576909 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp"] Oct 30 01:33:25.581199 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:33:25.581169 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod410ab4b3_ac20_4633_8311_763564d670e4.slice/crio-14c49e7c096217ca3f19939973775bfb9b030d1c9dd4128bfd45877c8c02cc0f WatchSource:0}: Error finding container 14c49e7c096217ca3f19939973775bfb9b030d1c9dd4128bfd45877c8c02cc0f: Status 404 returned error can't find the container with id 14c49e7c096217ca3f19939973775bfb9b030d1c9dd4128bfd45877c8c02cc0f Oct 30 01:33:25.583633 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:25.583601 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:33:25.583706 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:25.583668 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:33:25.583706 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:25.583699 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:33:26.301271 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:26.301240 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp" event={"ID":"410ab4b3-ac20-4633-8311-763564d670e4","Type":"ContainerStarted","Data":"9b32bb2c137f4eca7b288669b28dd98133d4cd6c1b2062aca640f3fe4b1d5fbb"} Oct 30 01:33:26.301719 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:26.301291 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp" event={"ID":"410ab4b3-ac20-4633-8311-763564d670e4","Type":"ContainerStarted","Data":"14c49e7c096217ca3f19939973775bfb9b030d1c9dd4128bfd45877c8c02cc0f"} Oct 30 01:33:26.376166 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:26.376118 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp" podStartSLOduration=2.376103832 podStartE2EDuration="2.376103832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:33:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:33:26.36123387 +0000 UTC m=+929.939521207" watchObservedRunningTime="2026-10-30 01:33:26.376103832 +0000 UTC m=+929.954391167" Oct 30 01:33:27.253002 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:27.252962 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp" Oct 30 01:33:27.254565 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:27.254513 6221 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 01:33:27.254679 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:27.254586 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp" podUID="410ab4b3-ac20-4633-8311-763564d670e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 01:33:27.340629 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:27.337394 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-278fe7d5/e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx"] Oct 30 01:33:27.342501 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:27.342479 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-278fe7d5/e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx" Oct 30 01:33:27.363123 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:27.363093 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-278fe7d5/e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx"] Oct 30 01:33:27.363274 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:27.363192 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-278fe7d5\"/\"kube-root-ca.crt\"" Oct 30 01:33:27.372374 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:27.372350 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-278fe7d5\"/\"openshift-service-ca.crt\"" Oct 30 01:33:27.372374 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:27.372368 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-278fe7d5\"/\"default-dockercfg-6rwpf\"" Oct 30 01:33:27.375044 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:27.375027 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-278fe7d5\"/\"e2e-lifecycle-model-278fe7d5-kserve-self-signed-certs\"" Oct 30 01:33:27.447729 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:27.447699 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03e0644c-f8df-4da7-9414-ff2bb32e2855-tmp-dir\") pod \"e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx\" (UID: \"03e0644c-f8df-4da7-9414-ff2bb32e2855\") " pod="ai-tenant-e2e-mt-278fe7d5/e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx" Oct 30 01:33:27.447902 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:27.447863 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03e0644c-f8df-4da7-9414-ff2bb32e2855-model-cache\") pod \"e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx\" (UID: \"03e0644c-f8df-4da7-9414-ff2bb32e2855\") " pod="ai-tenant-e2e-mt-278fe7d5/e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx" Oct 30 01:33:27.447989 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:27.447904 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03e0644c-f8df-4da7-9414-ff2bb32e2855-tls-certs\") pod \"e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx\" (UID: \"03e0644c-f8df-4da7-9414-ff2bb32e2855\") " pod="ai-tenant-e2e-mt-278fe7d5/e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx" Oct 30 01:33:27.447989 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:27.447974 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxcml\" (UniqueName: \"kubernetes.io/projected/03e0644c-f8df-4da7-9414-ff2bb32e2855-kube-api-access-rxcml\") pod \"e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx\" (UID: \"03e0644c-f8df-4da7-9414-ff2bb32e2855\") " pod="ai-tenant-e2e-mt-278fe7d5/e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx" Oct 30 01:33:27.448239 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:27.448221 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03e0644c-f8df-4da7-9414-ff2bb32e2855-dshm\") pod \"e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx\" (UID: \"03e0644c-f8df-4da7-9414-ff2bb32e2855\") " pod="ai-tenant-e2e-mt-278fe7d5/e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx" Oct 30 01:33:27.448352 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:27.448334 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03e0644c-f8df-4da7-9414-ff2bb32e2855-home\") pod \"e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx\" (UID: \"03e0644c-f8df-4da7-9414-ff2bb32e2855\") " pod="ai-tenant-e2e-mt-278fe7d5/e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx" Oct 30 01:33:27.549500 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:27.549463 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxcml\" (UniqueName: \"kubernetes.io/projected/03e0644c-f8df-4da7-9414-ff2bb32e2855-kube-api-access-rxcml\") pod \"e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx\" (UID: \"03e0644c-f8df-4da7-9414-ff2bb32e2855\") " pod="ai-tenant-e2e-mt-278fe7d5/e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx" Oct 30 01:33:27.549696 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:27.549639 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03e0644c-f8df-4da7-9414-ff2bb32e2855-dshm\") pod \"e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx\" (UID: \"03e0644c-f8df-4da7-9414-ff2bb32e2855\") " pod="ai-tenant-e2e-mt-278fe7d5/e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx" Oct 30 01:33:27.549799 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:27.549777 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03e0644c-f8df-4da7-9414-ff2bb32e2855-home\") pod \"e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx\" (UID: \"03e0644c-f8df-4da7-9414-ff2bb32e2855\") " pod="ai-tenant-e2e-mt-278fe7d5/e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx" Oct 30 01:33:27.549862 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:27.549833 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03e0644c-f8df-4da7-9414-ff2bb32e2855-tmp-dir\") pod \"e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx\" (UID: \"03e0644c-f8df-4da7-9414-ff2bb32e2855\") " pod="ai-tenant-e2e-mt-278fe7d5/e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx" Oct 30 01:33:27.549957 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:27.549943 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03e0644c-f8df-4da7-9414-ff2bb32e2855-model-cache\") pod \"e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx\" (UID: \"03e0644c-f8df-4da7-9414-ff2bb32e2855\") " pod="ai-tenant-e2e-mt-278fe7d5/e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx" Oct 30 01:33:27.550012 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:27.549971 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03e0644c-f8df-4da7-9414-ff2bb32e2855-tls-certs\") pod \"e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx\" (UID: \"03e0644c-f8df-4da7-9414-ff2bb32e2855\") " pod="ai-tenant-e2e-mt-278fe7d5/e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx" Oct 30 01:33:27.550120 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:27.550101 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03e0644c-f8df-4da7-9414-ff2bb32e2855-home\") pod \"e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx\" (UID: \"03e0644c-f8df-4da7-9414-ff2bb32e2855\") " pod="ai-tenant-e2e-mt-278fe7d5/e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx" Oct 30 01:33:27.550211 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:27.550183 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03e0644c-f8df-4da7-9414-ff2bb32e2855-tmp-dir\") pod \"e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx\" (UID: \"03e0644c-f8df-4da7-9414-ff2bb32e2855\") " pod="ai-tenant-e2e-mt-278fe7d5/e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx" Oct 30 01:33:27.550287 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:27.550269 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03e0644c-f8df-4da7-9414-ff2bb32e2855-model-cache\") pod \"e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx\" (UID: \"03e0644c-f8df-4da7-9414-ff2bb32e2855\") " pod="ai-tenant-e2e-mt-278fe7d5/e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx" Oct 30 01:33:27.551969 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:27.551946 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03e0644c-f8df-4da7-9414-ff2bb32e2855-dshm\") pod \"e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx\" (UID: \"03e0644c-f8df-4da7-9414-ff2bb32e2855\") " pod="ai-tenant-e2e-mt-278fe7d5/e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx" Oct 30 01:33:27.552377 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:27.552358 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03e0644c-f8df-4da7-9414-ff2bb32e2855-tls-certs\") pod \"e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx\" (UID: \"03e0644c-f8df-4da7-9414-ff2bb32e2855\") " pod="ai-tenant-e2e-mt-278fe7d5/e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx" Oct 30 01:33:27.632566 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:27.629612 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxcml\" (UniqueName: \"kubernetes.io/projected/03e0644c-f8df-4da7-9414-ff2bb32e2855-kube-api-access-rxcml\") pod \"e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx\" (UID: \"03e0644c-f8df-4da7-9414-ff2bb32e2855\") " pod="ai-tenant-e2e-mt-278fe7d5/e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx" Oct 30 01:33:27.673554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:27.672855 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-278fe7d5/e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx" Oct 30 01:33:28.098252 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:28.098224 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-278fe7d5/e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx"] Oct 30 01:33:28.100360 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:33:28.100321 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03e0644c_f8df_4da7_9414_ff2bb32e2855.slice/crio-f2abce721efccda7b5a28dcbe14ade3408558a8b9d74352c294c35d550e9af5d WatchSource:0}: Error finding container f2abce721efccda7b5a28dcbe14ade3408558a8b9d74352c294c35d550e9af5d: Status 404 returned error can't find the container with id f2abce721efccda7b5a28dcbe14ade3408558a8b9d74352c294c35d550e9af5d Oct 30 01:33:28.253129 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:28.253097 6221 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 01:33:28.253330 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:28.253158 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp" podUID="410ab4b3-ac20-4633-8311-763564d670e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 01:33:28.390612 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:28.389389 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-278fe7d5/e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx" event={"ID":"03e0644c-f8df-4da7-9414-ff2bb32e2855","Type":"ContainerStarted","Data":"3316e44d699d26a5cbb669f29bb323ea71235d206dfbe3d93770b3502d76cc4d"} Oct 30 01:33:28.390612 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:28.389448 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-278fe7d5/e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx" event={"ID":"03e0644c-f8df-4da7-9414-ff2bb32e2855","Type":"ContainerStarted","Data":"f2abce721efccda7b5a28dcbe14ade3408558a8b9d74352c294c35d550e9af5d"} Oct 30 01:33:28.437556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:28.436820 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-278fe7d5/e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx" podStartSLOduration=1.4368003200000001 podStartE2EDuration="1.43680032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:33:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:33:28.428374321 +0000 UTC m=+932.006661662" watchObservedRunningTime="2026-10-30 01:33:28.43680032 +0000 UTC m=+932.015087657" Oct 30 01:33:28.674211 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:28.674121 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-278fe7d5/e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx" Oct 30 01:33:29.253508 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:29.253475 6221 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 01:33:29.253746 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:29.253542 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp" podUID="410ab4b3-ac20-4633-8311-763564d670e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 01:33:29.878667 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:29.878634 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh"] Oct 30 01:33:29.882719 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:29.882697 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" Oct 30 01:33:29.935514 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:29.935471 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh"] Oct 30 01:33:29.938553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:29.936606 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-26hnh\" (UID: \"c9bddd88-6e7d-47fe-99b1-11c8208f3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" Oct 30 01:33:29.938553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:29.937750 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-26hnh\" (UID: \"c9bddd88-6e7d-47fe-99b1-11c8208f3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" Oct 30 01:33:29.938553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:29.937809 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49lp4\" (UniqueName: \"kubernetes.io/projected/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-kube-api-access-49lp4\") pod \"maas-default-gateway-openshift-default-5bc884dd87-26hnh\" (UID: \"c9bddd88-6e7d-47fe-99b1-11c8208f3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" Oct 30 01:33:29.938553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:29.937846 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-26hnh\" (UID: \"c9bddd88-6e7d-47fe-99b1-11c8208f3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" Oct 30 01:33:29.938553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:29.937874 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-26hnh\" (UID: \"c9bddd88-6e7d-47fe-99b1-11c8208f3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" Oct 30 01:33:29.938553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:29.937941 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-26hnh\" (UID: \"c9bddd88-6e7d-47fe-99b1-11c8208f3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" Oct 30 01:33:29.938553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:29.938106 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-26hnh\" (UID: \"c9bddd88-6e7d-47fe-99b1-11c8208f3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" Oct 30 01:33:29.938553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:29.938137 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-26hnh\" (UID: \"c9bddd88-6e7d-47fe-99b1-11c8208f3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" Oct 30 01:33:29.938553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:29.938230 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-26hnh\" (UID: \"c9bddd88-6e7d-47fe-99b1-11c8208f3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" Oct 30 01:33:30.039932 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:30.039894 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-26hnh\" (UID: \"c9bddd88-6e7d-47fe-99b1-11c8208f3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" Oct 30 01:33:30.040099 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:30.039943 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-26hnh\" (UID: \"c9bddd88-6e7d-47fe-99b1-11c8208f3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" Oct 30 01:33:30.040099 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:30.039984 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-26hnh\" (UID: \"c9bddd88-6e7d-47fe-99b1-11c8208f3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" Oct 30 01:33:30.040099 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:30.040010 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49lp4\" (UniqueName: \"kubernetes.io/projected/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-kube-api-access-49lp4\") pod \"maas-default-gateway-openshift-default-5bc884dd87-26hnh\" (UID: \"c9bddd88-6e7d-47fe-99b1-11c8208f3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" Oct 30 01:33:30.040099 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:30.040037 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-26hnh\" (UID: \"c9bddd88-6e7d-47fe-99b1-11c8208f3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" Oct 30 01:33:30.040099 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:30.040054 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-26hnh\" (UID: \"c9bddd88-6e7d-47fe-99b1-11c8208f3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" Oct 30 01:33:30.040410 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:30.040108 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-26hnh\" (UID: \"c9bddd88-6e7d-47fe-99b1-11c8208f3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" Oct 30 01:33:30.040410 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:30.040271 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-26hnh\" (UID: \"c9bddd88-6e7d-47fe-99b1-11c8208f3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" Oct 30 01:33:30.040410 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:30.040301 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-26hnh\" (UID: \"c9bddd88-6e7d-47fe-99b1-11c8208f3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" Oct 30 01:33:30.040596 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:30.040463 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-26hnh\" (UID: \"c9bddd88-6e7d-47fe-99b1-11c8208f3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" Oct 30 01:33:30.040596 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:30.040567 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-26hnh\" (UID: \"c9bddd88-6e7d-47fe-99b1-11c8208f3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" Oct 30 01:33:30.040806 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:30.040776 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-26hnh\" (UID: \"c9bddd88-6e7d-47fe-99b1-11c8208f3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" Oct 30 01:33:30.040806 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:30.040801 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-26hnh\" (UID: \"c9bddd88-6e7d-47fe-99b1-11c8208f3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" Oct 30 01:33:30.041144 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:30.041123 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-26hnh\" (UID: \"c9bddd88-6e7d-47fe-99b1-11c8208f3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" Oct 30 01:33:30.042526 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:30.042507 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-26hnh\" (UID: \"c9bddd88-6e7d-47fe-99b1-11c8208f3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" Oct 30 01:33:30.042773 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:30.042756 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-26hnh\" (UID: \"c9bddd88-6e7d-47fe-99b1-11c8208f3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" Oct 30 01:33:30.059171 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:30.059141 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-26hnh\" (UID: \"c9bddd88-6e7d-47fe-99b1-11c8208f3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" Oct 30 01:33:30.060372 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:30.060355 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49lp4\" (UniqueName: \"kubernetes.io/projected/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-kube-api-access-49lp4\") pod \"maas-default-gateway-openshift-default-5bc884dd87-26hnh\" (UID: \"c9bddd88-6e7d-47fe-99b1-11c8208f3fef\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" Oct 30 01:33:30.202024 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:30.201941 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" Oct 30 01:33:30.279593 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:30.279284 6221 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 01:33:30.279593 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:30.279347 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp" podUID="410ab4b3-ac20-4633-8311-763564d670e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 01:33:30.640868 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:30.640834 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh"] Oct 30 01:33:30.644824 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:33:30.644796 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9bddd88_6e7d_47fe_99b1_11c8208f3fef.slice/crio-3075f2de3899365594f657cbc184ca1da78069872cbf293f8eedeccf0b1e0c85 WatchSource:0}: Error finding container 3075f2de3899365594f657cbc184ca1da78069872cbf293f8eedeccf0b1e0c85: Status 404 returned error can't find the container with id 3075f2de3899365594f657cbc184ca1da78069872cbf293f8eedeccf0b1e0c85 Oct 30 01:33:30.647298 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:30.647265 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:33:30.647374 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:30.647331 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:33:30.647374 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:30.647362 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:33:31.252972 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:31.252933 6221 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 01:33:31.253375 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:31.252987 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp" podUID="410ab4b3-ac20-4633-8311-763564d670e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 01:33:31.434227 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:31.434178 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" event={"ID":"c9bddd88-6e7d-47fe-99b1-11c8208f3fef","Type":"ContainerStarted","Data":"3bcc7fc6e2ffd25b8cea2a3e14f29a57c274edd7dd09862e3613195eb373ff79"} Oct 30 01:33:31.434429 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:31.434249 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" event={"ID":"c9bddd88-6e7d-47fe-99b1-11c8208f3fef","Type":"ContainerStarted","Data":"3075f2de3899365594f657cbc184ca1da78069872cbf293f8eedeccf0b1e0c85"} Oct 30 01:33:31.501723 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:31.501673 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" podStartSLOduration=2.50165878 podStartE2EDuration="2.50165878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:33:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:33:31.487980792 +0000 UTC m=+935.066268125" watchObservedRunningTime="2026-10-30 01:33:31.50165878 +0000 UTC m=+935.079946115" Oct 30 01:33:32.202985 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:32.202948 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" Oct 30 01:33:32.204439 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:32.204414 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-26hnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 01:33:32.204561 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:32.204460 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" podUID="c9bddd88-6e7d-47fe-99b1-11c8208f3fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 01:33:32.252840 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:32.252800 6221 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 30 01:33:32.253026 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:32.252883 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp" podUID="410ab4b3-ac20-4633-8311-763564d670e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 30 01:33:33.202881 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:33.202842 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-26hnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 01:33:33.203052 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:33.202892 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" podUID="c9bddd88-6e7d-47fe-99b1-11c8208f3fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 01:33:33.244970 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:33.243455 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6"] Oct 30 01:33:33.249791 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:33.249760 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6" Oct 30 01:33:33.262802 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:33.262761 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6"] Oct 30 01:33:33.343981 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:33.343948 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp" Oct 30 01:33:33.344492 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:33.344468 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp" Oct 30 01:33:33.346741 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:33.346721 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp" Oct 30 01:33:33.428988 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:33.428953 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5499971-7479-4b2e-969c-e10ab43b95f1-istio-token\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6\" (UID: \"b5499971-7479-4b2e-969c-e10ab43b95f1\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6" Oct 30 01:33:33.429736 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:33.429707 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5499971-7479-4b2e-969c-e10ab43b95f1-istio-podinfo\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6\" (UID: \"b5499971-7479-4b2e-969c-e10ab43b95f1\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6" Oct 30 01:33:33.429978 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:33.429956 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5499971-7479-4b2e-969c-e10ab43b95f1-istio-envoy\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6\" (UID: \"b5499971-7479-4b2e-969c-e10ab43b95f1\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6" Oct 30 01:33:33.433501 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:33.433478 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5499971-7479-4b2e-969c-e10ab43b95f1-istio-data\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6\" (UID: \"b5499971-7479-4b2e-969c-e10ab43b95f1\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6" Oct 30 01:33:33.434697 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:33.434674 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5499971-7479-4b2e-969c-e10ab43b95f1-workload-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6\" (UID: \"b5499971-7479-4b2e-969c-e10ab43b95f1\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6" Oct 30 01:33:33.434924 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:33.434902 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5499971-7479-4b2e-969c-e10ab43b95f1-istiod-ca-cert\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6\" (UID: \"b5499971-7479-4b2e-969c-e10ab43b95f1\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6" Oct 30 01:33:33.435171 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:33.435151 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5499971-7479-4b2e-969c-e10ab43b95f1-workload-certs\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6\" (UID: \"b5499971-7479-4b2e-969c-e10ab43b95f1\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6" Oct 30 01:33:33.435342 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:33.435322 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5499971-7479-4b2e-969c-e10ab43b95f1-credential-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6\" (UID: \"b5499971-7479-4b2e-969c-e10ab43b95f1\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6" Oct 30 01:33:33.435462 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:33.435446 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd7zh\" (UniqueName: \"kubernetes.io/projected/b5499971-7479-4b2e-969c-e10ab43b95f1-kube-api-access-qd7zh\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6\" (UID: \"b5499971-7479-4b2e-969c-e10ab43b95f1\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6" Oct 30 01:33:33.537286 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:33.537249 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5499971-7479-4b2e-969c-e10ab43b95f1-workload-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6\" (UID: \"b5499971-7479-4b2e-969c-e10ab43b95f1\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6" Oct 30 01:33:33.537478 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:33.537293 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5499971-7479-4b2e-969c-e10ab43b95f1-istiod-ca-cert\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6\" (UID: \"b5499971-7479-4b2e-969c-e10ab43b95f1\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6" Oct 30 01:33:33.537478 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:33.537334 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5499971-7479-4b2e-969c-e10ab43b95f1-workload-certs\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6\" (UID: \"b5499971-7479-4b2e-969c-e10ab43b95f1\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6" Oct 30 01:33:33.537478 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:33.537414 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5499971-7479-4b2e-969c-e10ab43b95f1-credential-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6\" (UID: \"b5499971-7479-4b2e-969c-e10ab43b95f1\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6" Oct 30 01:33:33.537478 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:33.537437 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd7zh\" (UniqueName: \"kubernetes.io/projected/b5499971-7479-4b2e-969c-e10ab43b95f1-kube-api-access-qd7zh\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6\" (UID: \"b5499971-7479-4b2e-969c-e10ab43b95f1\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6" Oct 30 01:33:33.537724 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:33.537516 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5499971-7479-4b2e-969c-e10ab43b95f1-istio-token\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6\" (UID: \"b5499971-7479-4b2e-969c-e10ab43b95f1\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6" Oct 30 01:33:33.538039 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:33.538002 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5499971-7479-4b2e-969c-e10ab43b95f1-workload-certs\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6\" (UID: \"b5499971-7479-4b2e-969c-e10ab43b95f1\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6" Oct 30 01:33:33.538161 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:33.538066 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5499971-7479-4b2e-969c-e10ab43b95f1-istio-podinfo\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6\" (UID: \"b5499971-7479-4b2e-969c-e10ab43b95f1\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6" Oct 30 01:33:33.538222 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:33.538161 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5499971-7479-4b2e-969c-e10ab43b95f1-istio-envoy\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6\" (UID: \"b5499971-7479-4b2e-969c-e10ab43b95f1\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6" Oct 30 01:33:33.538293 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:33.538257 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5499971-7479-4b2e-969c-e10ab43b95f1-istio-data\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6\" (UID: \"b5499971-7479-4b2e-969c-e10ab43b95f1\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6" Oct 30 01:33:33.538346 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:33.538286 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5499971-7479-4b2e-969c-e10ab43b95f1-workload-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6\" (UID: \"b5499971-7479-4b2e-969c-e10ab43b95f1\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6" Oct 30 01:33:33.538422 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:33.538398 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5499971-7479-4b2e-969c-e10ab43b95f1-credential-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6\" (UID: \"b5499971-7479-4b2e-969c-e10ab43b95f1\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6" Oct 30 01:33:33.538782 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:33.538757 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5499971-7479-4b2e-969c-e10ab43b95f1-istio-data\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6\" (UID: \"b5499971-7479-4b2e-969c-e10ab43b95f1\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6" Oct 30 01:33:33.538862 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:33.538771 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5499971-7479-4b2e-969c-e10ab43b95f1-istiod-ca-cert\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6\" (UID: \"b5499971-7479-4b2e-969c-e10ab43b95f1\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6" Oct 30 01:33:33.540945 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:33.540926 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5499971-7479-4b2e-969c-e10ab43b95f1-istio-podinfo\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6\" (UID: \"b5499971-7479-4b2e-969c-e10ab43b95f1\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6" Oct 30 01:33:33.541250 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:33.541230 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5499971-7479-4b2e-969c-e10ab43b95f1-istio-envoy\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6\" (UID: \"b5499971-7479-4b2e-969c-e10ab43b95f1\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6" Oct 30 01:33:33.559278 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:33.559248 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5499971-7479-4b2e-969c-e10ab43b95f1-istio-token\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6\" (UID: \"b5499971-7479-4b2e-969c-e10ab43b95f1\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6" Oct 30 01:33:33.559414 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:33.559247 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd7zh\" (UniqueName: \"kubernetes.io/projected/b5499971-7479-4b2e-969c-e10ab43b95f1-kube-api-access-qd7zh\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6\" (UID: \"b5499971-7479-4b2e-969c-e10ab43b95f1\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6" Oct 30 01:33:33.585722 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:33.585667 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6" Oct 30 01:33:33.961025 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:33.960992 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6"] Oct 30 01:33:33.963055 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:33:33.963027 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5499971_7479_4b2e_969c_e10ab43b95f1.slice/crio-5fb53eedd0cc1a8d77f2a4069b857ef0f29a3edae4b2421d8473d292270a7a09 WatchSource:0}: Error finding container 5fb53eedd0cc1a8d77f2a4069b857ef0f29a3edae4b2421d8473d292270a7a09: Status 404 returned error can't find the container with id 5fb53eedd0cc1a8d77f2a4069b857ef0f29a3edae4b2421d8473d292270a7a09 Oct 30 01:33:33.965420 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:33.965382 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:33:33.965503 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:33.965475 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:33:33.965570 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:33.965504 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:33:34.203359 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:34.203326 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-26hnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 01:33:34.203522 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:34.203380 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" podUID="c9bddd88-6e7d-47fe-99b1-11c8208f3fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 01:33:34.471089 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:34.471047 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6" event={"ID":"b5499971-7479-4b2e-969c-e10ab43b95f1","Type":"ContainerStarted","Data":"5f1ed4126b73545b5f734a0337a9383cfb6471dfff05ffda627d642e9db9de1d"} Oct 30 01:33:34.471515 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:34.471115 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6" event={"ID":"b5499971-7479-4b2e-969c-e10ab43b95f1","Type":"ContainerStarted","Data":"5fb53eedd0cc1a8d77f2a4069b857ef0f29a3edae4b2421d8473d292270a7a09"} Oct 30 01:33:34.525562 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:34.525483 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6" podStartSLOduration=1.5254643479999999 podStartE2EDuration="1.525464348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:33:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:33:34.520948665 +0000 UTC m=+938.099236013" watchObservedRunningTime="2026-10-30 01:33:34.525464348 +0000 UTC m=+938.103751686" Oct 30 01:33:34.586325 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:34.586293 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6" Oct 30 01:33:35.202727 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:35.202692 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-26hnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 01:33:35.202976 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:35.202744 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" podUID="c9bddd88-6e7d-47fe-99b1-11c8208f3fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 01:33:35.600006 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:35.596746 6221 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 01:33:35.600006 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:35.596820 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6" podUID="b5499971-7479-4b2e-969c-e10ab43b95f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 01:33:36.202946 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:36.202901 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-26hnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 01:33:36.203158 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:36.202970 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" podUID="c9bddd88-6e7d-47fe-99b1-11c8208f3fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 01:33:36.275143 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:36.275103 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:33:36.586713 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:36.586677 6221 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 01:33:36.586899 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:36.586743 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6" podUID="b5499971-7479-4b2e-969c-e10ab43b95f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 01:33:37.215588 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:37.214461 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-26hnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 01:33:37.215588 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:37.214553 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" podUID="c9bddd88-6e7d-47fe-99b1-11c8208f3fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 01:33:37.594556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:37.593774 6221 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 01:33:37.594556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:37.593834 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6" podUID="b5499971-7479-4b2e-969c-e10ab43b95f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 01:33:37.678567 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:37.677009 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-278fe7d5/e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx" Oct 30 01:33:37.789556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:37.789444 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-278fe7d5/e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx" Oct 30 01:33:38.209558 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:38.203922 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-26hnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 01:33:38.209558 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:38.203991 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" podUID="c9bddd88-6e7d-47fe-99b1-11c8208f3fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 01:33:38.601555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:38.596328 6221 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 01:33:38.601555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:38.596400 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6" podUID="b5499971-7479-4b2e-969c-e10ab43b95f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 01:33:38.616562 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:38.615751 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" podUID="9df52a01-0c8d-44ba-9fd7-9cb2fa41947a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:38.725661 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:38.725611 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" podUID="354ec282-b972-4fca-b4e3-1e33ef461f2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:38.829705 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:38.829655 6221 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" podUID="018c3fe2-3e67-45ed-a3d7-d70f3fe5b956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:38.887729 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:38.887623 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" podUID="018c3fe2-3e67-45ed-a3d7-d70f3fe5b956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:38.927556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:38.926608 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" podUID="3b3c0e08-1371-4184-816a-d8b501526190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:39.086412 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:39.086064 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-278fe7d5/e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx" Oct 30 01:33:39.205019 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:39.204928 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-26hnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 01:33:39.238739 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:39.238700 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" podUID="c9bddd88-6e7d-47fe-99b1-11c8208f3fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 01:33:39.601554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:39.599638 6221 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 01:33:39.601554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:39.599703 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6" podUID="b5499971-7479-4b2e-969c-e10ab43b95f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 01:33:39.727567 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:39.727493 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" podUID="354ec282-b972-4fca-b4e3-1e33ef461f2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:39.728267 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:39.728244 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" Oct 30 01:33:39.948605 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:39.944623 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" podUID="3b3c0e08-1371-4184-816a-d8b501526190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:39.948605 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:39.946776 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" Oct 30 01:33:40.105921 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:40.105876 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" podUID="ea9583aa-8259-4c9b-932e-2d6559008d86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:40.214614 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:40.214500 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-26hnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 01:33:40.214856 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:40.214829 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" podUID="c9bddd88-6e7d-47fe-99b1-11c8208f3fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 01:33:40.284561 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:40.280633 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk" podUID="f4ae494e-81ec-4039-ae07-7ed1bf345f0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Oct 30 01:33:40.636557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:40.635793 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-278fe7d5/e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx" podUID="03e0644c-f8df-4da7-9414-ff2bb32e2855" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:40.687556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:40.675658 6221 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 01:33:40.687556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:40.675722 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6" podUID="b5499971-7479-4b2e-969c-e10ab43b95f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 01:33:40.940673 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:40.933693 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" podUID="354ec282-b972-4fca-b4e3-1e33ef461f2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 01:33:40.992440 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:40.941015 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" podUID="018c3fe2-3e67-45ed-a3d7-d70f3fe5b956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 01:33:40.992757 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:40.967885 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" podUID="3b3c0e08-1371-4184-816a-d8b501526190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:40.992960 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:40.967996 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" podUID="9df52a01-0c8d-44ba-9fd7-9cb2fa41947a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 01:33:41.119555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:41.116808 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" podUID="ea9583aa-8259-4c9b-932e-2d6559008d86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:41.220611 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:41.219991 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" Oct 30 01:33:41.359673 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:41.359623 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk" podUID="f4ae494e-81ec-4039-ae07-7ed1bf345f0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:41.385913 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:41.385879 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk" Oct 30 01:33:41.462944 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:41.462908 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-wvm5b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:33:41.490241 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:41.490154 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b" podUID="f33783d9-0aa2-4089-878d-ffc93fded2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:41.640196 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:41.640129 6221 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 01:33:41.680985 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:41.680944 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6" podUID="b5499971-7479-4b2e-969c-e10ab43b95f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 01:33:41.681686 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:41.646639 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-278fe7d5/e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx" podUID="03e0644c-f8df-4da7-9414-ff2bb32e2855" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded" Oct 30 01:33:41.682327 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:41.682305 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-278fe7d5/e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx" Oct 30 01:33:42.011800 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:42.011750 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" podUID="354ec282-b972-4fca-b4e3-1e33ef461f2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:42.030680 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:42.030631 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" podUID="9df52a01-0c8d-44ba-9fd7-9cb2fa41947a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:42.123556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:42.030917 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" podUID="3b3c0e08-1371-4184-816a-d8b501526190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 01:33:42.123556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:42.122713 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" Oct 30 01:33:42.123824 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:42.032454 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" podUID="018c3fe2-3e67-45ed-a3d7-d70f3fe5b956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 01:33:42.127553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:42.121779 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" podUID="ea9583aa-8259-4c9b-932e-2d6559008d86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:42.127553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:42.124183 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" Oct 30 01:33:42.315618 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:42.290798 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-26hnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:33:42.315618 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:42.290951 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" podUID="c9bddd88-6e7d-47fe-99b1-11c8208f3fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:42.380313 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:42.380281 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" Oct 30 01:33:42.454607 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:42.380748 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk" podUID="f4ae494e-81ec-4039-ae07-7ed1bf345f0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:33:42.679770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:42.679685 6221 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 01:33:42.680336 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:42.680304 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6" podUID="b5499971-7479-4b2e-969c-e10ab43b95f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 01:33:42.686285 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:42.686259 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" Oct 30 01:33:42.687244 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:42.687122 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-278fe7d5/e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx" Oct 30 01:33:42.687430 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:42.687414 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk" Oct 30 01:33:42.724060 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:42.724029 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" Oct 30 01:33:42.757019 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:42.756990 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" Oct 30 01:33:42.757269 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:42.757256 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" Oct 30 01:33:42.830890 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:42.830305 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" Oct 30 01:33:42.835689 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:42.835310 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" Oct 30 01:33:42.838437 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:42.838386 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" Oct 30 01:33:43.995622 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:43.995587 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6" Oct 30 01:33:43.997132 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:43.996326 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6" Oct 30 01:33:43.998433 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:43.998410 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6" Oct 30 01:33:51.575025 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:33:51.574966 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:33:58.065097 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:58.065062 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8"] Oct 30 01:33:58.069895 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:58.069869 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" Oct 30 01:33:58.093861 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:58.093820 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8"] Oct 30 01:33:58.161680 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:58.161647 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-4d38e5-openshift-default-dockercfg-xgzpg\"" Oct 30 01:33:58.167360 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:58.167332 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-istiod-ca-cert\") pod \"e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8\" (UID: \"bae879f9-b49c-4f58-a5b3-8c8dd936c89f\") " pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" Oct 30 01:33:58.167514 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:58.167373 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-istio-podinfo\") pod \"e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8\" (UID: \"bae879f9-b49c-4f58-a5b3-8c8dd936c89f\") " pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" Oct 30 01:33:58.167514 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:58.167397 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-istio-token\") pod \"e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8\" (UID: \"bae879f9-b49c-4f58-a5b3-8c8dd936c89f\") " pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" Oct 30 01:33:58.167608 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:58.167545 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-istio-data\") pod \"e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8\" (UID: \"bae879f9-b49c-4f58-a5b3-8c8dd936c89f\") " pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" Oct 30 01:33:58.167608 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:58.167592 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-workload-socket\") pod \"e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8\" (UID: \"bae879f9-b49c-4f58-a5b3-8c8dd936c89f\") " pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" Oct 30 01:33:58.167681 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:58.167653 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-istio-envoy\") pod \"e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8\" (UID: \"bae879f9-b49c-4f58-a5b3-8c8dd936c89f\") " pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" Oct 30 01:33:58.167738 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:58.167724 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-workload-certs\") pod \"e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8\" (UID: \"bae879f9-b49c-4f58-a5b3-8c8dd936c89f\") " pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" Oct 30 01:33:58.167849 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:58.167829 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgbcj\" (UniqueName: \"kubernetes.io/projected/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-kube-api-access-sgbcj\") pod \"e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8\" (UID: \"bae879f9-b49c-4f58-a5b3-8c8dd936c89f\") " pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" Oct 30 01:33:58.167921 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:58.167907 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-credential-socket\") pod \"e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8\" (UID: \"bae879f9-b49c-4f58-a5b3-8c8dd936c89f\") " pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" Oct 30 01:33:58.276281 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:58.276241 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgbcj\" (UniqueName: \"kubernetes.io/projected/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-kube-api-access-sgbcj\") pod \"e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8\" (UID: \"bae879f9-b49c-4f58-a5b3-8c8dd936c89f\") " pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" Oct 30 01:33:58.277728 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:58.277703 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-credential-socket\") pod \"e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8\" (UID: \"bae879f9-b49c-4f58-a5b3-8c8dd936c89f\") " pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" Oct 30 01:33:58.278274 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:58.278258 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-istiod-ca-cert\") pod \"e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8\" (UID: \"bae879f9-b49c-4f58-a5b3-8c8dd936c89f\") " pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" Oct 30 01:33:58.279059 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:58.279042 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-istio-podinfo\") pod \"e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8\" (UID: \"bae879f9-b49c-4f58-a5b3-8c8dd936c89f\") " pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" Oct 30 01:33:58.285925 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:58.285899 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-istio-token\") pod \"e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8\" (UID: \"bae879f9-b49c-4f58-a5b3-8c8dd936c89f\") " pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" Oct 30 01:33:58.302764 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:58.302731 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-istio-data\") pod \"e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8\" (UID: \"bae879f9-b49c-4f58-a5b3-8c8dd936c89f\") " pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" Oct 30 01:33:58.303343 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:58.303320 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-workload-socket\") pod \"e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8\" (UID: \"bae879f9-b49c-4f58-a5b3-8c8dd936c89f\") " pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" Oct 30 01:33:58.303510 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:58.303494 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-istio-envoy\") pod \"e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8\" (UID: \"bae879f9-b49c-4f58-a5b3-8c8dd936c89f\") " pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" Oct 30 01:33:58.304269 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:58.304246 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-workload-certs\") pod \"e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8\" (UID: \"bae879f9-b49c-4f58-a5b3-8c8dd936c89f\") " pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" Oct 30 01:33:58.304583 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:58.303886 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-workload-socket\") pod \"e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8\" (UID: \"bae879f9-b49c-4f58-a5b3-8c8dd936c89f\") " pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" Oct 30 01:33:58.304720 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:58.278182 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-credential-socket\") pod \"e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8\" (UID: \"bae879f9-b49c-4f58-a5b3-8c8dd936c89f\") " pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" Oct 30 01:33:58.304823 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:58.278986 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-istiod-ca-cert\") pod \"e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8\" (UID: \"bae879f9-b49c-4f58-a5b3-8c8dd936c89f\") " pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" Oct 30 01:33:58.304918 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:58.303240 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-istio-data\") pod \"e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8\" (UID: \"bae879f9-b49c-4f58-a5b3-8c8dd936c89f\") " pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" Oct 30 01:33:58.305281 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:58.305257 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-workload-certs\") pod \"e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8\" (UID: \"bae879f9-b49c-4f58-a5b3-8c8dd936c89f\") " pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" Oct 30 01:33:58.311362 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:58.311337 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgbcj\" (UniqueName: \"kubernetes.io/projected/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-kube-api-access-sgbcj\") pod \"e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8\" (UID: \"bae879f9-b49c-4f58-a5b3-8c8dd936c89f\") " pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" Oct 30 01:33:58.337592 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:58.337110 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-istio-podinfo\") pod \"e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8\" (UID: \"bae879f9-b49c-4f58-a5b3-8c8dd936c89f\") " pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" Oct 30 01:33:58.348552 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:58.338037 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-istio-token\") pod \"e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8\" (UID: \"bae879f9-b49c-4f58-a5b3-8c8dd936c89f\") " pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" Oct 30 01:33:58.348552 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:58.339419 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-istio-envoy\") pod \"e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8\" (UID: \"bae879f9-b49c-4f58-a5b3-8c8dd936c89f\") " pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" Oct 30 01:33:58.394091 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:58.393804 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" Oct 30 01:33:58.756265 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:58.756238 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8"] Oct 30 01:33:58.758832 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:33:58.758805 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbae879f9_b49c_4f58_a5b3_8c8dd936c89f.slice/crio-120d823e18c013becbbdbc5b8dd910cff277766a87723680afa7fbc4895e1fc6 WatchSource:0}: Error finding container 120d823e18c013becbbdbc5b8dd910cff277766a87723680afa7fbc4895e1fc6: Status 404 returned error can't find the container with id 120d823e18c013becbbdbc5b8dd910cff277766a87723680afa7fbc4895e1fc6 Oct 30 01:33:58.761177 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:58.761148 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:33:58.761242 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:58.761214 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:33:58.761279 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:58.761242 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:33:59.657013 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:59.656976 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" event={"ID":"bae879f9-b49c-4f58-a5b3-8c8dd936c89f","Type":"ContainerStarted","Data":"b1210ea546ae404f24da2f0b41fa282e95b95c9575a194d71891569b6c632fc0"} Oct 30 01:33:59.657442 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:59.657031 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" event={"ID":"bae879f9-b49c-4f58-a5b3-8c8dd936c89f","Type":"ContainerStarted","Data":"120d823e18c013becbbdbc5b8dd910cff277766a87723680afa7fbc4895e1fc6"} Oct 30 01:33:59.750553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:33:59.749753 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" podStartSLOduration=1.74973315 podStartE2EDuration="1.74973315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:33:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:33:59.730123095 +0000 UTC m=+963.308410432" watchObservedRunningTime="2026-10-30 01:33:59.74973315 +0000 UTC m=+963.328020487" Oct 30 01:34:00.404554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:00.395134 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" Oct 30 01:34:00.409620 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:00.409587 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" Oct 30 01:34:00.703886 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:00.703804 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" Oct 30 01:34:00.732551 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:00.728854 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" Oct 30 01:34:04.079267 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:04.079230 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4d38e5-775c67c59-wrshf"] Oct 30 01:34:04.084474 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:04.084455 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4d38e5-775c67c59-wrshf" Oct 30 01:34:04.102304 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:04.102270 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4d38e5-775c67c59-wrshf"] Oct 30 01:34:04.108844 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:04.108802 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32b0bea5-05bb-4c7a-942d-eb057e672324-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4d38e5-775c67c59-wrshf\" (UID: \"32b0bea5-05bb-4c7a-942d-eb057e672324\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4d38e5-775c67c59-wrshf" Oct 30 01:34:04.109022 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:04.108917 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4mbc\" (UniqueName: \"kubernetes.io/projected/32b0bea5-05bb-4c7a-942d-eb057e672324-kube-api-access-z4mbc\") pod \"payload-pre-processing-e2e-api-b-4d38e5-775c67c59-wrshf\" (UID: \"32b0bea5-05bb-4c7a-942d-eb057e672324\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4d38e5-775c67c59-wrshf" Oct 30 01:34:04.117825 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:04.117795 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-4d38e5\"" Oct 30 01:34:04.118823 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:04.118763 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-4d38e5-dockercfg-wqgc5\"" Oct 30 01:34:04.226550 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:04.221925 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32b0bea5-05bb-4c7a-942d-eb057e672324-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4d38e5-775c67c59-wrshf\" (UID: \"32b0bea5-05bb-4c7a-942d-eb057e672324\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4d38e5-775c67c59-wrshf" Oct 30 01:34:04.226550 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:04.226367 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4mbc\" (UniqueName: \"kubernetes.io/projected/32b0bea5-05bb-4c7a-942d-eb057e672324-kube-api-access-z4mbc\") pod \"payload-pre-processing-e2e-api-b-4d38e5-775c67c59-wrshf\" (UID: \"32b0bea5-05bb-4c7a-942d-eb057e672324\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4d38e5-775c67c59-wrshf" Oct 30 01:34:04.231372 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:04.228239 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32b0bea5-05bb-4c7a-942d-eb057e672324-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4d38e5-775c67c59-wrshf\" (UID: \"32b0bea5-05bb-4c7a-942d-eb057e672324\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4d38e5-775c67c59-wrshf" Oct 30 01:34:04.302714 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:04.300040 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4d38e5-569dc98886-flmzc"] Oct 30 01:34:04.317596 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:04.307328 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4d38e5-569dc98886-flmzc"] Oct 30 01:34:04.317596 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:04.307485 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4d38e5-569dc98886-flmzc" Oct 30 01:34:04.371232 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:04.353390 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfl9x\" (UniqueName: \"kubernetes.io/projected/bffcc9b8-9193-4110-950c-fc16323075e8-kube-api-access-wfl9x\") pod \"payload-processing-e2e-api-b-4d38e5-569dc98886-flmzc\" (UID: \"bffcc9b8-9193-4110-950c-fc16323075e8\") " pod="openshift-ingress/payload-processing-e2e-api-b-4d38e5-569dc98886-flmzc" Oct 30 01:34:04.371232 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:04.353472 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bffcc9b8-9193-4110-950c-fc16323075e8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4d38e5-569dc98886-flmzc\" (UID: \"bffcc9b8-9193-4110-950c-fc16323075e8\") " pod="openshift-ingress/payload-processing-e2e-api-b-4d38e5-569dc98886-flmzc" Oct 30 01:34:04.401551 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:04.394883 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4mbc\" (UniqueName: \"kubernetes.io/projected/32b0bea5-05bb-4c7a-942d-eb057e672324-kube-api-access-z4mbc\") pod \"payload-pre-processing-e2e-api-b-4d38e5-775c67c59-wrshf\" (UID: \"32b0bea5-05bb-4c7a-942d-eb057e672324\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4d38e5-775c67c59-wrshf" Oct 30 01:34:04.414379 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:04.403601 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4d38e5-775c67c59-wrshf" Oct 30 01:34:04.455740 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:04.455699 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfl9x\" (UniqueName: \"kubernetes.io/projected/bffcc9b8-9193-4110-950c-fc16323075e8-kube-api-access-wfl9x\") pod \"payload-processing-e2e-api-b-4d38e5-569dc98886-flmzc\" (UID: \"bffcc9b8-9193-4110-950c-fc16323075e8\") " pod="openshift-ingress/payload-processing-e2e-api-b-4d38e5-569dc98886-flmzc" Oct 30 01:34:04.455887 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:04.455761 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bffcc9b8-9193-4110-950c-fc16323075e8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4d38e5-569dc98886-flmzc\" (UID: \"bffcc9b8-9193-4110-950c-fc16323075e8\") " pod="openshift-ingress/payload-processing-e2e-api-b-4d38e5-569dc98886-flmzc" Oct 30 01:34:04.457736 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:04.457714 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bffcc9b8-9193-4110-950c-fc16323075e8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4d38e5-569dc98886-flmzc\" (UID: \"bffcc9b8-9193-4110-950c-fc16323075e8\") " pod="openshift-ingress/payload-processing-e2e-api-b-4d38e5-569dc98886-flmzc" Oct 30 01:34:04.476178 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:04.476149 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfl9x\" (UniqueName: \"kubernetes.io/projected/bffcc9b8-9193-4110-950c-fc16323075e8-kube-api-access-wfl9x\") pod \"payload-processing-e2e-api-b-4d38e5-569dc98886-flmzc\" (UID: \"bffcc9b8-9193-4110-950c-fc16323075e8\") " pod="openshift-ingress/payload-processing-e2e-api-b-4d38e5-569dc98886-flmzc" Oct 30 01:34:04.480572 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:04.480543 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4d38e5-569dc98886-flmzc" Oct 30 01:34:04.757123 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:04.757095 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4d38e5-775c67c59-wrshf"] Oct 30 01:34:04.758942 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:34:04.758916 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32b0bea5_05bb_4c7a_942d_eb057e672324.slice/crio-0b8e167908fa1eeb0fb095a47f96f2f7b504651b22b878d50332e9b0b06dce0b WatchSource:0}: Error finding container 0b8e167908fa1eeb0fb095a47f96f2f7b504651b22b878d50332e9b0b06dce0b: Status 404 returned error can't find the container with id 0b8e167908fa1eeb0fb095a47f96f2f7b504651b22b878d50332e9b0b06dce0b Oct 30 01:34:04.795438 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:04.795416 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4d38e5-569dc98886-flmzc"] Oct 30 01:34:04.798029 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:34:04.798004 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbffcc9b8_9193_4110_950c_fc16323075e8.slice/crio-42b72d2ca8d2e819005d73f0385266730416860d0e3b53e2717e5f6d7e7aa01d WatchSource:0}: Error finding container 42b72d2ca8d2e819005d73f0385266730416860d0e3b53e2717e5f6d7e7aa01d: Status 404 returned error can't find the container with id 42b72d2ca8d2e819005d73f0385266730416860d0e3b53e2717e5f6d7e7aa01d Oct 30 01:34:05.768004 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:05.767972 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4d38e5-775c67c59-wrshf" event={"ID":"32b0bea5-05bb-4c7a-942d-eb057e672324","Type":"ContainerStarted","Data":"5e130e589f41ef91d95ba72f7874e9e1a20b651c58e0b7a5eac7d98198d062bf"} Oct 30 01:34:05.768458 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:05.768023 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4d38e5-775c67c59-wrshf" event={"ID":"32b0bea5-05bb-4c7a-942d-eb057e672324","Type":"ContainerStarted","Data":"0b8e167908fa1eeb0fb095a47f96f2f7b504651b22b878d50332e9b0b06dce0b"} Oct 30 01:34:05.768458 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:05.768071 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4d38e5-775c67c59-wrshf" Oct 30 01:34:05.769649 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:05.769627 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4d38e5-569dc98886-flmzc" event={"ID":"bffcc9b8-9193-4110-950c-fc16323075e8","Type":"ContainerStarted","Data":"6945d0238df51b9d8c253f006d0ca33af12b469359684574d489c4972ad1a832"} Oct 30 01:34:05.769762 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:05.769661 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4d38e5-569dc98886-flmzc" event={"ID":"bffcc9b8-9193-4110-950c-fc16323075e8","Type":"ContainerStarted","Data":"42b72d2ca8d2e819005d73f0385266730416860d0e3b53e2717e5f6d7e7aa01d"} Oct 30 01:34:05.769762 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:05.769739 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-4d38e5-569dc98886-flmzc" Oct 30 01:34:05.887732 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:05.887679 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4d38e5-775c67c59-wrshf" podStartSLOduration=1.887666093 podStartE2EDuration="1.887666093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:05.819884499 +0000 UTC m=+969.398171835" watchObservedRunningTime="2026-10-30 01:34:05.887666093 +0000 UTC m=+969.465953448" Oct 30 01:34:05.888382 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:05.888348 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-4d38e5-569dc98886-flmzc" podStartSLOduration=1.888338417 podStartE2EDuration="1.888338417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:05.885618557 +0000 UTC m=+969.463905893" watchObservedRunningTime="2026-10-30 01:34:05.888338417 +0000 UTC m=+969.466625752" Oct 30 01:34:16.778889 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:16.778860 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4d38e5-775c67c59-wrshf" Oct 30 01:34:16.779346 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:16.778937 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-4d38e5-569dc98886-flmzc" Oct 30 01:34:17.902020 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:17.901985 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a06822-8557d79667-ctpsb"] Oct 30 01:34:17.905900 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:17.905884 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a06822-8557d79667-ctpsb" Oct 30 01:34:17.926800 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:17.926780 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-a06822\"" Oct 30 01:34:17.926902 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:17.926816 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-a06822-dockercfg-8ss6z\"" Oct 30 01:34:17.944051 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:17.944023 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a06822-8557d79667-ctpsb"] Oct 30 01:34:18.029765 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.023766 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvwd5\" (UniqueName: \"kubernetes.io/projected/1200b6f0-d740-4ca0-80b6-3fa083ce7156-kube-api-access-rvwd5\") pod \"payload-pre-processing-e2e-api-a-a06822-8557d79667-ctpsb\" (UID: \"1200b6f0-d740-4ca0-80b6-3fa083ce7156\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a06822-8557d79667-ctpsb" Oct 30 01:34:18.037723 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.037682 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1200b6f0-d740-4ca0-80b6-3fa083ce7156-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a06822-8557d79667-ctpsb\" (UID: \"1200b6f0-d740-4ca0-80b6-3fa083ce7156\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a06822-8557d79667-ctpsb" Oct 30 01:34:18.075425 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.075380 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a06822-7c4d8ddf5b-xtpl4"] Oct 30 01:34:18.109975 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.109944 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a06822-7c4d8ddf5b-xtpl4" Oct 30 01:34:18.126115 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.126087 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a06822-7c4d8ddf5b-xtpl4"] Oct 30 01:34:18.153094 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.152997 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84b9ecd9-6438-40ee-be8b-23558589777f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a06822-7c4d8ddf5b-xtpl4\" (UID: \"84b9ecd9-6438-40ee-be8b-23558589777f\") " pod="openshift-ingress/payload-processing-e2e-api-a-a06822-7c4d8ddf5b-xtpl4" Oct 30 01:34:18.153386 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.153363 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvwd5\" (UniqueName: \"kubernetes.io/projected/1200b6f0-d740-4ca0-80b6-3fa083ce7156-kube-api-access-rvwd5\") pod \"payload-pre-processing-e2e-api-a-a06822-8557d79667-ctpsb\" (UID: \"1200b6f0-d740-4ca0-80b6-3fa083ce7156\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a06822-8557d79667-ctpsb" Oct 30 01:34:18.153579 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.153561 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1200b6f0-d740-4ca0-80b6-3fa083ce7156-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a06822-8557d79667-ctpsb\" (UID: \"1200b6f0-d740-4ca0-80b6-3fa083ce7156\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a06822-8557d79667-ctpsb" Oct 30 01:34:18.153833 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.153814 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmwnq\" (UniqueName: \"kubernetes.io/projected/84b9ecd9-6438-40ee-be8b-23558589777f-kube-api-access-qmwnq\") pod \"payload-processing-e2e-api-a-a06822-7c4d8ddf5b-xtpl4\" (UID: \"84b9ecd9-6438-40ee-be8b-23558589777f\") " pod="openshift-ingress/payload-processing-e2e-api-a-a06822-7c4d8ddf5b-xtpl4" Oct 30 01:34:18.155299 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.155273 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1200b6f0-d740-4ca0-80b6-3fa083ce7156-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a06822-8557d79667-ctpsb\" (UID: \"1200b6f0-d740-4ca0-80b6-3fa083ce7156\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a06822-8557d79667-ctpsb" Oct 30 01:34:18.229642 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.229575 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvwd5\" (UniqueName: \"kubernetes.io/projected/1200b6f0-d740-4ca0-80b6-3fa083ce7156-kube-api-access-rvwd5\") pod \"payload-pre-processing-e2e-api-a-a06822-8557d79667-ctpsb\" (UID: \"1200b6f0-d740-4ca0-80b6-3fa083ce7156\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a06822-8557d79667-ctpsb" Oct 30 01:34:18.261979 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.255445 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmwnq\" (UniqueName: \"kubernetes.io/projected/84b9ecd9-6438-40ee-be8b-23558589777f-kube-api-access-qmwnq\") pod \"payload-processing-e2e-api-a-a06822-7c4d8ddf5b-xtpl4\" (UID: \"84b9ecd9-6438-40ee-be8b-23558589777f\") " pod="openshift-ingress/payload-processing-e2e-api-a-a06822-7c4d8ddf5b-xtpl4" Oct 30 01:34:18.261979 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.255585 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84b9ecd9-6438-40ee-be8b-23558589777f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a06822-7c4d8ddf5b-xtpl4\" (UID: \"84b9ecd9-6438-40ee-be8b-23558589777f\") " pod="openshift-ingress/payload-processing-e2e-api-a-a06822-7c4d8ddf5b-xtpl4" Oct 30 01:34:18.269079 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.269042 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84b9ecd9-6438-40ee-be8b-23558589777f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a06822-7c4d8ddf5b-xtpl4\" (UID: \"84b9ecd9-6438-40ee-be8b-23558589777f\") " pod="openshift-ingress/payload-processing-e2e-api-a-a06822-7c4d8ddf5b-xtpl4" Oct 30 01:34:18.287704 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.287672 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmwnq\" (UniqueName: \"kubernetes.io/projected/84b9ecd9-6438-40ee-be8b-23558589777f-kube-api-access-qmwnq\") pod \"payload-processing-e2e-api-a-a06822-7c4d8ddf5b-xtpl4\" (UID: \"84b9ecd9-6438-40ee-be8b-23558589777f\") " pod="openshift-ingress/payload-processing-e2e-api-a-a06822-7c4d8ddf5b-xtpl4" Oct 30 01:34:18.363189 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.363147 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb"] Oct 30 01:34:18.368547 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.368515 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb" Oct 30 01:34:18.385096 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.385064 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb"] Oct 30 01:34:18.393454 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.393424 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb"] Oct 30 01:34:18.394846 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.393550 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb" Oct 30 01:34:18.408572 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.408503 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb"] Oct 30 01:34:18.446262 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.446222 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a06822-7c4d8ddf5b-xtpl4" Oct 30 01:34:18.457848 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.457818 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d56da67-6cfc-49c4-802b-3c9579c87110-workload-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb\" (UID: \"0d56da67-6cfc-49c4-802b-3c9579c87110\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb" Oct 30 01:34:18.457991 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.457864 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-istiod-ca-cert\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb\" (UID: \"0ef9c920-60ba-43ce-a4a2-2f5bc2731279\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb" Oct 30 01:34:18.458055 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.458004 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfv65\" (UniqueName: \"kubernetes.io/projected/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-kube-api-access-qfv65\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb\" (UID: \"0ef9c920-60ba-43ce-a4a2-2f5bc2731279\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb" Oct 30 01:34:18.458293 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.458098 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d56da67-6cfc-49c4-802b-3c9579c87110-istio-envoy\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb\" (UID: \"0d56da67-6cfc-49c4-802b-3c9579c87110\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb" Oct 30 01:34:18.458293 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.458131 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d56da67-6cfc-49c4-802b-3c9579c87110-istiod-ca-cert\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb\" (UID: \"0d56da67-6cfc-49c4-802b-3c9579c87110\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb" Oct 30 01:34:18.458293 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.458165 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-workload-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb\" (UID: \"0ef9c920-60ba-43ce-a4a2-2f5bc2731279\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb" Oct 30 01:34:18.458498 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.458282 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d56da67-6cfc-49c4-802b-3c9579c87110-credential-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb\" (UID: \"0d56da67-6cfc-49c4-802b-3c9579c87110\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb" Oct 30 01:34:18.458498 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.458329 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-istio-token\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb\" (UID: \"0ef9c920-60ba-43ce-a4a2-2f5bc2731279\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb" Oct 30 01:34:18.458498 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.458425 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-credential-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb\" (UID: \"0ef9c920-60ba-43ce-a4a2-2f5bc2731279\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb" Oct 30 01:34:18.458498 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.458456 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d56da67-6cfc-49c4-802b-3c9579c87110-istio-token\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb\" (UID: \"0d56da67-6cfc-49c4-802b-3c9579c87110\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb" Oct 30 01:34:18.458728 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.458497 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqdqw\" (UniqueName: \"kubernetes.io/projected/0d56da67-6cfc-49c4-802b-3c9579c87110-kube-api-access-zqdqw\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb\" (UID: \"0d56da67-6cfc-49c4-802b-3c9579c87110\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb" Oct 30 01:34:18.458728 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.458681 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-istio-envoy\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb\" (UID: \"0ef9c920-60ba-43ce-a4a2-2f5bc2731279\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb" Oct 30 01:34:18.458839 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.458735 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-workload-certs\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb\" (UID: \"0ef9c920-60ba-43ce-a4a2-2f5bc2731279\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb" Oct 30 01:34:18.458839 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.458763 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d56da67-6cfc-49c4-802b-3c9579c87110-istio-podinfo\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb\" (UID: \"0d56da67-6cfc-49c4-802b-3c9579c87110\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb" Oct 30 01:34:18.458839 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.458786 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d56da67-6cfc-49c4-802b-3c9579c87110-workload-certs\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb\" (UID: \"0d56da67-6cfc-49c4-802b-3c9579c87110\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb" Oct 30 01:34:18.458989 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.458879 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d56da67-6cfc-49c4-802b-3c9579c87110-istio-data\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb\" (UID: \"0d56da67-6cfc-49c4-802b-3c9579c87110\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb" Oct 30 01:34:18.458989 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.458913 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-istio-data\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb\" (UID: \"0ef9c920-60ba-43ce-a4a2-2f5bc2731279\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb" Oct 30 01:34:18.458989 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.458947 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-istio-podinfo\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb\" (UID: \"0ef9c920-60ba-43ce-a4a2-2f5bc2731279\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb" Oct 30 01:34:18.523085 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.520119 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a06822-8557d79667-ctpsb" Oct 30 01:34:18.560195 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.560162 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-workload-certs\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb\" (UID: \"0ef9c920-60ba-43ce-a4a2-2f5bc2731279\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb" Oct 30 01:34:18.560374 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.560201 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d56da67-6cfc-49c4-802b-3c9579c87110-istio-podinfo\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb\" (UID: \"0d56da67-6cfc-49c4-802b-3c9579c87110\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb" Oct 30 01:34:18.560374 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.560229 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d56da67-6cfc-49c4-802b-3c9579c87110-workload-certs\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb\" (UID: \"0d56da67-6cfc-49c4-802b-3c9579c87110\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb" Oct 30 01:34:18.560374 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.560286 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d56da67-6cfc-49c4-802b-3c9579c87110-istio-data\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb\" (UID: \"0d56da67-6cfc-49c4-802b-3c9579c87110\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb" Oct 30 01:34:18.560374 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.560315 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-istio-data\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb\" (UID: \"0ef9c920-60ba-43ce-a4a2-2f5bc2731279\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb" Oct 30 01:34:18.560374 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.560351 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-istio-podinfo\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb\" (UID: \"0ef9c920-60ba-43ce-a4a2-2f5bc2731279\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb" Oct 30 01:34:18.560656 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.560458 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d56da67-6cfc-49c4-802b-3c9579c87110-workload-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb\" (UID: \"0d56da67-6cfc-49c4-802b-3c9579c87110\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb" Oct 30 01:34:18.560656 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.560513 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-istiod-ca-cert\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb\" (UID: \"0ef9c920-60ba-43ce-a4a2-2f5bc2731279\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb" Oct 30 01:34:18.560656 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.560611 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfv65\" (UniqueName: \"kubernetes.io/projected/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-kube-api-access-qfv65\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb\" (UID: \"0ef9c920-60ba-43ce-a4a2-2f5bc2731279\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb" Oct 30 01:34:18.560833 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.560745 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d56da67-6cfc-49c4-802b-3c9579c87110-istio-envoy\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb\" (UID: \"0d56da67-6cfc-49c4-802b-3c9579c87110\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb" Oct 30 01:34:18.560833 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.560802 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d56da67-6cfc-49c4-802b-3c9579c87110-istiod-ca-cert\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb\" (UID: \"0d56da67-6cfc-49c4-802b-3c9579c87110\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb" Oct 30 01:34:18.560944 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.560868 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-workload-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb\" (UID: \"0ef9c920-60ba-43ce-a4a2-2f5bc2731279\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb" Oct 30 01:34:18.560944 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.560905 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d56da67-6cfc-49c4-802b-3c9579c87110-credential-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb\" (UID: \"0d56da67-6cfc-49c4-802b-3c9579c87110\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb" Oct 30 01:34:18.561057 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.560956 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-istio-token\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb\" (UID: \"0ef9c920-60ba-43ce-a4a2-2f5bc2731279\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb" Oct 30 01:34:18.561117 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.561065 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-credential-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb\" (UID: \"0ef9c920-60ba-43ce-a4a2-2f5bc2731279\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb" Oct 30 01:34:18.561183 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.561116 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d56da67-6cfc-49c4-802b-3c9579c87110-istio-token\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb\" (UID: \"0d56da67-6cfc-49c4-802b-3c9579c87110\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb" Oct 30 01:34:18.561241 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.561187 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqdqw\" (UniqueName: \"kubernetes.io/projected/0d56da67-6cfc-49c4-802b-3c9579c87110-kube-api-access-zqdqw\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb\" (UID: \"0d56da67-6cfc-49c4-802b-3c9579c87110\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb" Oct 30 01:34:18.561241 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.561218 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-istio-envoy\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb\" (UID: \"0ef9c920-60ba-43ce-a4a2-2f5bc2731279\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb" Oct 30 01:34:18.561350 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.561296 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-istiod-ca-cert\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb\" (UID: \"0ef9c920-60ba-43ce-a4a2-2f5bc2731279\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb" Oct 30 01:34:18.561642 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.561618 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-workload-certs\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb\" (UID: \"0ef9c920-60ba-43ce-a4a2-2f5bc2731279\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb" Oct 30 01:34:18.561805 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.561784 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-istio-data\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb\" (UID: \"0ef9c920-60ba-43ce-a4a2-2f5bc2731279\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb" Oct 30 01:34:18.562922 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.562523 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d56da67-6cfc-49c4-802b-3c9579c87110-istio-data\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb\" (UID: \"0d56da67-6cfc-49c4-802b-3c9579c87110\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb" Oct 30 01:34:18.563030 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.562869 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d56da67-6cfc-49c4-802b-3c9579c87110-workload-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb\" (UID: \"0d56da67-6cfc-49c4-802b-3c9579c87110\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb" Oct 30 01:34:18.563030 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.562919 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d56da67-6cfc-49c4-802b-3c9579c87110-workload-certs\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb\" (UID: \"0d56da67-6cfc-49c4-802b-3c9579c87110\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb" Oct 30 01:34:18.563030 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.562923 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-workload-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb\" (UID: \"0ef9c920-60ba-43ce-a4a2-2f5bc2731279\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb" Oct 30 01:34:18.563631 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.563611 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d56da67-6cfc-49c4-802b-3c9579c87110-istiod-ca-cert\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb\" (UID: \"0d56da67-6cfc-49c4-802b-3c9579c87110\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb" Oct 30 01:34:18.564107 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.563726 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-credential-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb\" (UID: \"0ef9c920-60ba-43ce-a4a2-2f5bc2731279\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb" Oct 30 01:34:18.564107 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.564031 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d56da67-6cfc-49c4-802b-3c9579c87110-credential-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb\" (UID: \"0d56da67-6cfc-49c4-802b-3c9579c87110\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb" Oct 30 01:34:18.564793 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.564627 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-istio-envoy\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb\" (UID: \"0ef9c920-60ba-43ce-a4a2-2f5bc2731279\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb" Oct 30 01:34:18.567559 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.566209 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-istio-podinfo\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb\" (UID: \"0ef9c920-60ba-43ce-a4a2-2f5bc2731279\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb" Oct 30 01:34:18.596901 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.596300 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqdqw\" (UniqueName: \"kubernetes.io/projected/0d56da67-6cfc-49c4-802b-3c9579c87110-kube-api-access-zqdqw\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb\" (UID: \"0d56da67-6cfc-49c4-802b-3c9579c87110\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb" Oct 30 01:34:18.606989 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.601261 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d56da67-6cfc-49c4-802b-3c9579c87110-istio-token\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb\" (UID: \"0d56da67-6cfc-49c4-802b-3c9579c87110\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb" Oct 30 01:34:18.606989 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.606889 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-istio-token\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb\" (UID: \"0ef9c920-60ba-43ce-a4a2-2f5bc2731279\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb" Oct 30 01:34:18.646124 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.641465 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfv65\" (UniqueName: \"kubernetes.io/projected/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-kube-api-access-qfv65\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb\" (UID: \"0ef9c920-60ba-43ce-a4a2-2f5bc2731279\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb" Oct 30 01:34:18.679095 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.678723 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d56da67-6cfc-49c4-802b-3c9579c87110-istio-envoy\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb\" (UID: \"0d56da67-6cfc-49c4-802b-3c9579c87110\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb" Oct 30 01:34:18.679835 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.679481 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d56da67-6cfc-49c4-802b-3c9579c87110-istio-podinfo\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb\" (UID: \"0d56da67-6cfc-49c4-802b-3c9579c87110\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb" Oct 30 01:34:18.700404 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.700213 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb" Oct 30 01:34:18.718502 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.718310 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb" Oct 30 01:34:18.770016 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.768782 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a06822-7c4d8ddf5b-xtpl4"] Oct 30 01:34:18.870571 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.870526 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a06822-8557d79667-ctpsb"] Oct 30 01:34:18.897141 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.897114 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a06822-7c4d8ddf5b-xtpl4" event={"ID":"84b9ecd9-6438-40ee-be8b-23558589777f","Type":"ContainerStarted","Data":"fc6ea9c74d6ddb7f4b17ea43d3e4b7083eff33a335baaa529d27ecb921330b56"} Oct 30 01:34:18.897240 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.897172 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a06822-7c4d8ddf5b-xtpl4" event={"ID":"84b9ecd9-6438-40ee-be8b-23558589777f","Type":"ContainerStarted","Data":"cb83807bd45e4531a90bec4bf006aa1df06c550b6d9e21916034c722a803c40b"} Oct 30 01:34:18.898063 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.898043 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-a06822-7c4d8ddf5b-xtpl4" Oct 30 01:34:18.899933 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.899905 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a06822-8557d79667-ctpsb" event={"ID":"1200b6f0-d740-4ca0-80b6-3fa083ce7156","Type":"ContainerStarted","Data":"b8543f9726e769eca8e8e0ae58069b502828b8a210cd0278c39191f3e753365b"} Oct 30 01:34:18.942814 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:18.942769 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-a06822-7c4d8ddf5b-xtpl4" podStartSLOduration=1.942756638 podStartE2EDuration="1.942756638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:18.938613305 +0000 UTC m=+982.516900652" watchObservedRunningTime="2026-10-30 01:34:18.942756638 +0000 UTC m=+982.521043975" Oct 30 01:34:19.035288 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:19.035259 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb"] Oct 30 01:34:19.037960 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:34:19.037931 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ef9c920_60ba_43ce_a4a2_2f5bc2731279.slice/crio-6f39d90e3d1fdba71a1d2d74287f8936be9230d4c78ce0cc0e2743f1c6dd0ded WatchSource:0}: Error finding container 6f39d90e3d1fdba71a1d2d74287f8936be9230d4c78ce0cc0e2743f1c6dd0ded: Status 404 returned error can't find the container with id 6f39d90e3d1fdba71a1d2d74287f8936be9230d4c78ce0cc0e2743f1c6dd0ded Oct 30 01:34:19.040517 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:19.040487 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:34:19.040624 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:19.040561 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:34:19.040624 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:19.040595 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:34:19.089893 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:19.087399 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb"] Oct 30 01:34:19.099609 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:19.091167 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:34:19.099609 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:19.091269 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:34:19.099609 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:19.091319 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:34:19.747561 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:19.747451 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:34:19.944585 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:19.937612 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb" event={"ID":"0d56da67-6cfc-49c4-802b-3c9579c87110","Type":"ContainerStarted","Data":"cdc620cdba9036461218c8eaf7939e0e3a185372e9ba03380aeac0cbfed1e20f"} Oct 30 01:34:19.944585 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:19.937678 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb" event={"ID":"0d56da67-6cfc-49c4-802b-3c9579c87110","Type":"ContainerStarted","Data":"ed63c0509ef13385eb9d4679a602ac9acaca1b3bf15f3dbe766e23f7bfdc0b8b"} Oct 30 01:34:19.944585 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:19.943445 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a06822-8557d79667-ctpsb" event={"ID":"1200b6f0-d740-4ca0-80b6-3fa083ce7156","Type":"ContainerStarted","Data":"da5860ed0d0929168af90d74c3091720936ff6a378965781880fe3c52c4dc7f6"} Oct 30 01:34:19.944585 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:19.943504 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a06822-8557d79667-ctpsb" Oct 30 01:34:19.949688 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:19.949489 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb" event={"ID":"0ef9c920-60ba-43ce-a4a2-2f5bc2731279","Type":"ContainerStarted","Data":"25a46734f3ed22e9662fc8e7fdf7d563f4c419cccfa061acdc1347852a075f2b"} Oct 30 01:34:19.949688 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:19.949575 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb" event={"ID":"0ef9c920-60ba-43ce-a4a2-2f5bc2731279","Type":"ContainerStarted","Data":"6f39d90e3d1fdba71a1d2d74287f8936be9230d4c78ce0cc0e2743f1c6dd0ded"} Oct 30 01:34:19.985692 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:19.985611 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb" podStartSLOduration=1.9855986460000001 podStartE2EDuration="1.985598646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:19.984450892 +0000 UTC m=+983.562738230" watchObservedRunningTime="2026-10-30 01:34:19.985598646 +0000 UTC m=+983.563885981" Oct 30 01:34:20.031614 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:20.031266 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a06822-8557d79667-ctpsb" podStartSLOduration=3.031244798 podStartE2EDuration="3.031244798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:20.028781147 +0000 UTC m=+983.607068485" watchObservedRunningTime="2026-10-30 01:34:20.031244798 +0000 UTC m=+983.609532136" Oct 30 01:34:20.085328 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:20.085277 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb" podStartSLOduration=2.085262256 podStartE2EDuration="2.085262256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:20.081526508 +0000 UTC m=+983.659813843" watchObservedRunningTime="2026-10-30 01:34:20.085262256 +0000 UTC m=+983.663549626" Oct 30 01:34:20.704679 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:20.704612 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb" Oct 30 01:34:20.706216 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:20.706189 6221 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 01:34:20.706489 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:20.706255 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb" podUID="0ef9c920-60ba-43ce-a4a2-2f5bc2731279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 01:34:20.727380 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:20.727290 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb" Oct 30 01:34:20.740565 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:20.738651 6221 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 01:34:20.740565 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:20.738715 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb" podUID="0d56da67-6cfc-49c4-802b-3c9579c87110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 01:34:21.500923 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:34:21.500871 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:34:21.701300 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:21.701273 6221 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 01:34:21.701670 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:21.701644 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb" podUID="0ef9c920-60ba-43ce-a4a2-2f5bc2731279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 01:34:21.721314 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:21.721282 6221 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 01:34:21.721615 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:21.721584 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb" podUID="0d56da67-6cfc-49c4-802b-3c9579c87110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 01:34:22.700909 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:22.700877 6221 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 01:34:22.701311 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:22.700933 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb" podUID="0ef9c920-60ba-43ce-a4a2-2f5bc2731279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 01:34:22.729554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:22.723877 6221 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 01:34:22.729554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:22.723932 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb" podUID="0d56da67-6cfc-49c4-802b-3c9579c87110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 01:34:23.700705 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:23.700670 6221 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 01:34:23.700888 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:23.700724 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb" podUID="0ef9c920-60ba-43ce-a4a2-2f5bc2731279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 01:34:23.721666 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:23.721630 6221 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 01:34:23.722751 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:23.722719 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb" podUID="0d56da67-6cfc-49c4-802b-3c9579c87110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 01:34:24.701090 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:24.701057 6221 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 01:34:24.701278 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:24.701107 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb" podUID="0ef9c920-60ba-43ce-a4a2-2f5bc2731279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 01:34:24.733621 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:24.733584 6221 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 01:34:24.734287 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:24.734245 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb" podUID="0d56da67-6cfc-49c4-802b-3c9579c87110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 01:34:25.700664 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:25.700627 6221 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 01:34:25.700861 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:25.700686 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb" podUID="0ef9c920-60ba-43ce-a4a2-2f5bc2731279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 01:34:25.719285 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:25.719256 6221 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 01:34:25.719462 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:25.719304 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb" podUID="0d56da67-6cfc-49c4-802b-3c9579c87110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 01:34:27.748657 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:27.748620 6221 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:34:27.749300 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:27.749269 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb" podUID="0ef9c920-60ba-43ce-a4a2-2f5bc2731279" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:34:27.761880 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:27.760775 6221 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:34:27.761880 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:27.760825 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb" podUID="0d56da67-6cfc-49c4-802b-3c9579c87110" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:34:28.304811 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:28.304778 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb" Oct 30 01:34:28.305578 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:28.305523 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb" Oct 30 01:34:28.306283 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:28.306263 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb" Oct 30 01:34:28.328257 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:28.328230 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb" Oct 30 01:34:28.328589 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:28.328566 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb" Oct 30 01:34:28.329420 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:28.329393 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb" Oct 30 01:34:31.029081 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:31.029049 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-a06822-7c4d8ddf5b-xtpl4" Oct 30 01:34:31.029523 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:31.029117 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a06822-8557d79667-ctpsb" Oct 30 01:34:41.769099 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:41.769021 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-wxkcd"] Oct 30 01:34:46.769612 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:46.769565 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-wxkcd" podUID="41211dad-aa70-47ff-b6bb-12426581ab00" containerName="payload-pre-processing" containerID="cri-o://465226a307196f73a364d6d233cfd00a766f70de1bcf8c822661103ef0647471" gracePeriod=25 Oct 30 01:34:51.539151 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:34:51.539110 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:34:54.010002 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:54.009969 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-wxkcd" podUID="41211dad-aa70-47ff-b6bb-12426581ab00" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:34:54.398952 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:54.398917 6221 generic.go:356] "Generic (PLEG): container finished" podID="bae879f9-b49c-4f58-a5b3-8c8dd936c89f" containerID="b1210ea546ae404f24da2f0b41fa282e95b95c9575a194d71891569b6c632fc0" exitCode=0 Oct 30 01:34:54.399131 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:54.399006 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" event={"ID":"bae879f9-b49c-4f58-a5b3-8c8dd936c89f","Type":"ContainerDied","Data":"b1210ea546ae404f24da2f0b41fa282e95b95c9575a194d71891569b6c632fc0"} Oct 30 01:34:54.399442 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:54.399427 6221 scope.go:122] "RemoveContainer" containerID="b1210ea546ae404f24da2f0b41fa282e95b95c9575a194d71891569b6c632fc0" Oct 30 01:34:54.400956 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:54.400933 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:34:54.401031 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:54.400994 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:34:54.401031 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:54.401027 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:34:55.419569 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:55.406828 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" event={"ID":"bae879f9-b49c-4f58-a5b3-8c8dd936c89f","Type":"ContainerStarted","Data":"b81e88138f11f719f0c85a765be028d12c35d1ed664c86c3726c5f5411ac04f5"} Oct 30 01:34:56.395361 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:56.395327 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" Oct 30 01:34:56.396313 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:56.396284 6221 patch_prober.go:29] interesting pod/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 01:34:56.396443 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:56.396341 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" podUID="bae879f9-b49c-4f58-a5b3-8c8dd936c89f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 01:34:57.395179 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:57.395143 6221 patch_prober.go:29] interesting pod/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 01:34:57.395603 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:57.395195 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" podUID="bae879f9-b49c-4f58-a5b3-8c8dd936c89f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 01:34:57.885121 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:57.885078 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf"] Oct 30 01:34:57.891447 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:57.891417 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" Oct 30 01:34:57.896290 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:57.896262 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf"] Oct 30 01:34:57.939709 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:57.939672 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-97c137-openshift-default-dockercfg-wrj2w\"" Oct 30 01:34:57.997603 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:57.991576 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d30042d8-3e81-4825-9afa-9848e0f06141-workload-certs\") pod \"e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf\" (UID: \"d30042d8-3e81-4825-9afa-9848e0f06141\") " pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" Oct 30 01:34:57.997603 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:57.991747 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d30042d8-3e81-4825-9afa-9848e0f06141-istio-data\") pod \"e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf\" (UID: \"d30042d8-3e81-4825-9afa-9848e0f06141\") " pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" Oct 30 01:34:57.997603 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:57.992044 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d30042d8-3e81-4825-9afa-9848e0f06141-istiod-ca-cert\") pod \"e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf\" (UID: \"d30042d8-3e81-4825-9afa-9848e0f06141\") " pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" Oct 30 01:34:57.997603 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:57.992099 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30042d8-3e81-4825-9afa-9848e0f06141-workload-socket\") pod \"e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf\" (UID: \"d30042d8-3e81-4825-9afa-9848e0f06141\") " pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" Oct 30 01:34:57.997603 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:57.992184 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d30042d8-3e81-4825-9afa-9848e0f06141-istio-token\") pod \"e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf\" (UID: \"d30042d8-3e81-4825-9afa-9848e0f06141\") " pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" Oct 30 01:34:57.997603 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:57.992232 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbrbq\" (UniqueName: \"kubernetes.io/projected/d30042d8-3e81-4825-9afa-9848e0f06141-kube-api-access-jbrbq\") pod \"e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf\" (UID: \"d30042d8-3e81-4825-9afa-9848e0f06141\") " pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" Oct 30 01:34:57.997603 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:57.992275 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30042d8-3e81-4825-9afa-9848e0f06141-credential-socket\") pod \"e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf\" (UID: \"d30042d8-3e81-4825-9afa-9848e0f06141\") " pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" Oct 30 01:34:57.997603 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:57.992307 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d30042d8-3e81-4825-9afa-9848e0f06141-istio-podinfo\") pod \"e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf\" (UID: \"d30042d8-3e81-4825-9afa-9848e0f06141\") " pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" Oct 30 01:34:57.997603 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:57.992351 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d30042d8-3e81-4825-9afa-9848e0f06141-istio-envoy\") pod \"e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf\" (UID: \"d30042d8-3e81-4825-9afa-9848e0f06141\") " pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" Oct 30 01:34:58.093673 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:58.093628 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d30042d8-3e81-4825-9afa-9848e0f06141-istiod-ca-cert\") pod \"e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf\" (UID: \"d30042d8-3e81-4825-9afa-9848e0f06141\") " pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" Oct 30 01:34:58.094959 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:58.094933 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30042d8-3e81-4825-9afa-9848e0f06141-workload-socket\") pod \"e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf\" (UID: \"d30042d8-3e81-4825-9afa-9848e0f06141\") " pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" Oct 30 01:34:58.117841 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:58.117804 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d30042d8-3e81-4825-9afa-9848e0f06141-istio-token\") pod \"e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf\" (UID: \"d30042d8-3e81-4825-9afa-9848e0f06141\") " pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" Oct 30 01:34:58.118127 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:58.118103 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbrbq\" (UniqueName: \"kubernetes.io/projected/d30042d8-3e81-4825-9afa-9848e0f06141-kube-api-access-jbrbq\") pod \"e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf\" (UID: \"d30042d8-3e81-4825-9afa-9848e0f06141\") " pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" Oct 30 01:34:58.118287 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:58.118272 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30042d8-3e81-4825-9afa-9848e0f06141-credential-socket\") pod \"e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf\" (UID: \"d30042d8-3e81-4825-9afa-9848e0f06141\") " pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" Oct 30 01:34:58.118402 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:58.118388 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d30042d8-3e81-4825-9afa-9848e0f06141-istio-podinfo\") pod \"e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf\" (UID: \"d30042d8-3e81-4825-9afa-9848e0f06141\") " pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" Oct 30 01:34:58.120149 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:58.120125 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d30042d8-3e81-4825-9afa-9848e0f06141-istio-envoy\") pod \"e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf\" (UID: \"d30042d8-3e81-4825-9afa-9848e0f06141\") " pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" Oct 30 01:34:58.120344 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:58.120326 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d30042d8-3e81-4825-9afa-9848e0f06141-workload-certs\") pod \"e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf\" (UID: \"d30042d8-3e81-4825-9afa-9848e0f06141\") " pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" Oct 30 01:34:58.120490 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:58.120475 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d30042d8-3e81-4825-9afa-9848e0f06141-istio-data\") pod \"e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf\" (UID: \"d30042d8-3e81-4825-9afa-9848e0f06141\") " pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" Oct 30 01:34:58.125335 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:58.125310 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d30042d8-3e81-4825-9afa-9848e0f06141-istio-data\") pod \"e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf\" (UID: \"d30042d8-3e81-4825-9afa-9848e0f06141\") " pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" Oct 30 01:34:58.125511 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:58.095407 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30042d8-3e81-4825-9afa-9848e0f06141-workload-socket\") pod \"e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf\" (UID: \"d30042d8-3e81-4825-9afa-9848e0f06141\") " pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" Oct 30 01:34:58.145235 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:58.094840 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d30042d8-3e81-4825-9afa-9848e0f06141-istiod-ca-cert\") pod \"e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf\" (UID: \"d30042d8-3e81-4825-9afa-9848e0f06141\") " pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" Oct 30 01:34:58.145826 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:58.133204 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d30042d8-3e81-4825-9afa-9848e0f06141-workload-certs\") pod \"e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf\" (UID: \"d30042d8-3e81-4825-9afa-9848e0f06141\") " pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" Oct 30 01:34:58.145972 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:58.119326 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30042d8-3e81-4825-9afa-9848e0f06141-credential-socket\") pod \"e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf\" (UID: \"d30042d8-3e81-4825-9afa-9848e0f06141\") " pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" Oct 30 01:34:58.158553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:58.155026 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d30042d8-3e81-4825-9afa-9848e0f06141-istio-envoy\") pod \"e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf\" (UID: \"d30042d8-3e81-4825-9afa-9848e0f06141\") " pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" Oct 30 01:34:58.159256 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:58.159228 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d30042d8-3e81-4825-9afa-9848e0f06141-istio-token\") pod \"e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf\" (UID: \"d30042d8-3e81-4825-9afa-9848e0f06141\") " pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" Oct 30 01:34:58.160266 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:58.160244 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbrbq\" (UniqueName: \"kubernetes.io/projected/d30042d8-3e81-4825-9afa-9848e0f06141-kube-api-access-jbrbq\") pod \"e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf\" (UID: \"d30042d8-3e81-4825-9afa-9848e0f06141\") " pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" Oct 30 01:34:58.188552 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:58.181221 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d30042d8-3e81-4825-9afa-9848e0f06141-istio-podinfo\") pod \"e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf\" (UID: \"d30042d8-3e81-4825-9afa-9848e0f06141\") " pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" Oct 30 01:34:58.223877 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:58.223838 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" Oct 30 01:34:58.395736 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:58.395653 6221 patch_prober.go:29] interesting pod/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 01:34:58.396314 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:58.396272 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" podUID="bae879f9-b49c-4f58-a5b3-8c8dd936c89f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 01:34:58.735663 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:58.735629 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf"] Oct 30 01:34:58.738866 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:34:58.738840 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd30042d8_3e81_4825_9afa_9848e0f06141.slice/crio-7f057a72629b8834b12e3b49dee6bdcb74da2312efbb44e50aaf0f3489c141c6 WatchSource:0}: Error finding container 7f057a72629b8834b12e3b49dee6bdcb74da2312efbb44e50aaf0f3489c141c6: Status 404 returned error can't find the container with id 7f057a72629b8834b12e3b49dee6bdcb74da2312efbb44e50aaf0f3489c141c6 Oct 30 01:34:58.742135 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:58.742100 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:34:58.742238 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:58.742175 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:34:58.742238 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:58.742201 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:34:58.905985 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:58.905944 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm"] Oct 30 01:34:58.921439 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:58.921399 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" Oct 30 01:34:58.938155 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:58.938124 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-288c8f-openshift-default-dockercfg-xvrts\"" Oct 30 01:34:58.942592 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:58.942566 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm"] Oct 30 01:34:59.016608 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:59.014359 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60112bce-f58b-467c-8afe-11a1bb92f891-credential-socket\") pod \"e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm\" (UID: \"60112bce-f58b-467c-8afe-11a1bb92f891\") " pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" Oct 30 01:34:59.016608 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:59.014425 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60112bce-f58b-467c-8afe-11a1bb92f891-istiod-ca-cert\") pod \"e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm\" (UID: \"60112bce-f58b-467c-8afe-11a1bb92f891\") " pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" Oct 30 01:34:59.016608 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:59.014479 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60112bce-f58b-467c-8afe-11a1bb92f891-workload-certs\") pod \"e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm\" (UID: \"60112bce-f58b-467c-8afe-11a1bb92f891\") " pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" Oct 30 01:34:59.016608 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:59.014661 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jv8v\" (UniqueName: \"kubernetes.io/projected/60112bce-f58b-467c-8afe-11a1bb92f891-kube-api-access-4jv8v\") pod \"e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm\" (UID: \"60112bce-f58b-467c-8afe-11a1bb92f891\") " pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" Oct 30 01:34:59.016608 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:59.014693 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60112bce-f58b-467c-8afe-11a1bb92f891-istio-podinfo\") pod \"e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm\" (UID: \"60112bce-f58b-467c-8afe-11a1bb92f891\") " pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" Oct 30 01:34:59.016608 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:59.014759 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60112bce-f58b-467c-8afe-11a1bb92f891-istio-envoy\") pod \"e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm\" (UID: \"60112bce-f58b-467c-8afe-11a1bb92f891\") " pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" Oct 30 01:34:59.016608 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:59.014816 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60112bce-f58b-467c-8afe-11a1bb92f891-istio-data\") pod \"e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm\" (UID: \"60112bce-f58b-467c-8afe-11a1bb92f891\") " pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" Oct 30 01:34:59.016608 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:59.014851 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60112bce-f58b-467c-8afe-11a1bb92f891-istio-token\") pod \"e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm\" (UID: \"60112bce-f58b-467c-8afe-11a1bb92f891\") " pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" Oct 30 01:34:59.016608 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:59.014906 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60112bce-f58b-467c-8afe-11a1bb92f891-workload-socket\") pod \"e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm\" (UID: \"60112bce-f58b-467c-8afe-11a1bb92f891\") " pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" Oct 30 01:34:59.159570 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:59.158882 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60112bce-f58b-467c-8afe-11a1bb92f891-workload-socket\") pod \"e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm\" (UID: \"60112bce-f58b-467c-8afe-11a1bb92f891\") " pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" Oct 30 01:34:59.159570 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:59.159475 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60112bce-f58b-467c-8afe-11a1bb92f891-credential-socket\") pod \"e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm\" (UID: \"60112bce-f58b-467c-8afe-11a1bb92f891\") " pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" Oct 30 01:34:59.159570 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:59.159526 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60112bce-f58b-467c-8afe-11a1bb92f891-istiod-ca-cert\") pod \"e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm\" (UID: \"60112bce-f58b-467c-8afe-11a1bb92f891\") " pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" Oct 30 01:34:59.159883 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:59.159607 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60112bce-f58b-467c-8afe-11a1bb92f891-workload-certs\") pod \"e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm\" (UID: \"60112bce-f58b-467c-8afe-11a1bb92f891\") " pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" Oct 30 01:34:59.159883 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:59.159814 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jv8v\" (UniqueName: \"kubernetes.io/projected/60112bce-f58b-467c-8afe-11a1bb92f891-kube-api-access-4jv8v\") pod \"e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm\" (UID: \"60112bce-f58b-467c-8afe-11a1bb92f891\") " pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" Oct 30 01:34:59.159883 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:59.159844 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60112bce-f58b-467c-8afe-11a1bb92f891-istio-podinfo\") pod \"e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm\" (UID: \"60112bce-f58b-467c-8afe-11a1bb92f891\") " pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" Oct 30 01:34:59.160050 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:59.159922 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60112bce-f58b-467c-8afe-11a1bb92f891-istio-envoy\") pod \"e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm\" (UID: \"60112bce-f58b-467c-8afe-11a1bb92f891\") " pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" Oct 30 01:34:59.160050 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:59.160000 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60112bce-f58b-467c-8afe-11a1bb92f891-istio-data\") pod \"e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm\" (UID: \"60112bce-f58b-467c-8afe-11a1bb92f891\") " pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" Oct 30 01:34:59.160050 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:59.160039 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60112bce-f58b-467c-8afe-11a1bb92f891-istio-token\") pod \"e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm\" (UID: \"60112bce-f58b-467c-8afe-11a1bb92f891\") " pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" Oct 30 01:34:59.161778 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:59.159362 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60112bce-f58b-467c-8afe-11a1bb92f891-workload-socket\") pod \"e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm\" (UID: \"60112bce-f58b-467c-8afe-11a1bb92f891\") " pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" Oct 30 01:34:59.161778 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:59.160780 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60112bce-f58b-467c-8afe-11a1bb92f891-credential-socket\") pod \"e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm\" (UID: \"60112bce-f58b-467c-8afe-11a1bb92f891\") " pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" Oct 30 01:34:59.163903 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:59.163865 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60112bce-f58b-467c-8afe-11a1bb92f891-istio-data\") pod \"e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm\" (UID: \"60112bce-f58b-467c-8afe-11a1bb92f891\") " pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" Oct 30 01:34:59.189478 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:59.178685 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60112bce-f58b-467c-8afe-11a1bb92f891-istiod-ca-cert\") pod \"e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm\" (UID: \"60112bce-f58b-467c-8afe-11a1bb92f891\") " pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" Oct 30 01:34:59.189478 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:59.184245 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60112bce-f58b-467c-8afe-11a1bb92f891-workload-certs\") pod \"e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm\" (UID: \"60112bce-f58b-467c-8afe-11a1bb92f891\") " pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" Oct 30 01:34:59.198077 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:59.198040 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60112bce-f58b-467c-8afe-11a1bb92f891-istio-podinfo\") pod \"e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm\" (UID: \"60112bce-f58b-467c-8afe-11a1bb92f891\") " pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" Oct 30 01:34:59.288086 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:59.281842 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60112bce-f58b-467c-8afe-11a1bb92f891-istio-envoy\") pod \"e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm\" (UID: \"60112bce-f58b-467c-8afe-11a1bb92f891\") " pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" Oct 30 01:34:59.288086 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:59.282578 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jv8v\" (UniqueName: \"kubernetes.io/projected/60112bce-f58b-467c-8afe-11a1bb92f891-kube-api-access-4jv8v\") pod \"e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm\" (UID: \"60112bce-f58b-467c-8afe-11a1bb92f891\") " pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" Oct 30 01:34:59.301823 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:59.294214 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60112bce-f58b-467c-8afe-11a1bb92f891-istio-token\") pod \"e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm\" (UID: \"60112bce-f58b-467c-8afe-11a1bb92f891\") " pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" Oct 30 01:34:59.385279 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:59.385242 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" Oct 30 01:34:59.395664 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:59.395634 6221 patch_prober.go:29] interesting pod/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 01:34:59.395786 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:59.395687 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" podUID="bae879f9-b49c-4f58-a5b3-8c8dd936c89f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 01:34:59.649588 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:59.640721 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" event={"ID":"d30042d8-3e81-4825-9afa-9848e0f06141","Type":"ContainerStarted","Data":"3249149ef74bd2d4da3f251e78605238a18942e5d5a4982ee8bc30f368553b2d"} Oct 30 01:34:59.649588 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:59.640792 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" event={"ID":"d30042d8-3e81-4825-9afa-9848e0f06141","Type":"ContainerStarted","Data":"7f057a72629b8834b12e3b49dee6bdcb74da2312efbb44e50aaf0f3489c141c6"} Oct 30 01:34:59.690315 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:59.690254 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" podStartSLOduration=2.690235381 podStartE2EDuration="2.690235381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:34:59.688119759 +0000 UTC m=+1023.266407099" watchObservedRunningTime="2026-10-30 01:34:59.690235381 +0000 UTC m=+1023.268522721" Oct 30 01:34:59.895655 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:59.888074 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm"] Oct 30 01:34:59.940555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:59.914725 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:34:59.940555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:59.914821 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:34:59.940555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:34:59.914865 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:35:00.231826 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:00.231735 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" Oct 30 01:35:00.237192 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:00.237162 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" Oct 30 01:35:00.394959 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:00.394921 6221 patch_prober.go:29] interesting pod/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 30 01:35:00.395133 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:00.394976 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" podUID="bae879f9-b49c-4f58-a5b3-8c8dd936c89f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 30 01:35:00.648926 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:00.648893 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" event={"ID":"60112bce-f58b-467c-8afe-11a1bb92f891","Type":"ContainerStarted","Data":"13cfd9a06c7ca72616f6a22f8316aa683814cc70af429cc4e1542bae0d310df5"} Oct 30 01:35:00.649470 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:00.648949 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" event={"ID":"60112bce-f58b-467c-8afe-11a1bb92f891","Type":"ContainerStarted","Data":"2e4bbf9f532372379e3a9fcdd6e801369703908f1f06bec44e9291c4c2c545c8"} Oct 30 01:35:00.649470 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:00.649285 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" Oct 30 01:35:00.650185 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:00.650165 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" Oct 30 01:35:00.693523 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:00.693459 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" podStartSLOduration=2.693441903 podStartE2EDuration="2.693441903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:34:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:35:00.690348345 +0000 UTC m=+1024.268635682" watchObservedRunningTime="2026-10-30 01:35:00.693441903 +0000 UTC m=+1024.271729241" Oct 30 01:35:00.704039 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:00.704009 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" Oct 30 01:35:01.386206 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:01.386173 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" Oct 30 01:35:01.392368 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:01.392343 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" Oct 30 01:35:01.399519 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:01.399498 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" Oct 30 01:35:01.656915 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:01.656824 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" Oct 30 01:35:01.657635 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:01.657614 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" Oct 30 01:35:01.657764 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:01.657667 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" Oct 30 01:35:02.643198 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:02.643162 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-x4w45"] Oct 30 01:35:02.643494 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:02.643460 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-x4w45" podUID="5371c07a-6563-4db1-9483-32e9300a0581" containerName="maas-api" containerID="cri-o://7a749af26f034e0988a412a349f35edba89574946bb69cfe7380931e74f63330" gracePeriod=30 Oct 30 01:35:03.085553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:03.082457 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-x4w45" Oct 30 01:35:03.149554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:03.145170 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5371c07a-6563-4db1-9483-32e9300a0581-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5371c07a-6563-4db1-9483-32e9300a0581-metrics-certs\") pod \"5371c07a-6563-4db1-9483-32e9300a0581\" (UID: \"5371c07a-6563-4db1-9483-32e9300a0581\") " Oct 30 01:35:03.149554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:03.145271 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5371c07a-6563-4db1-9483-32e9300a0581-kube-api-access-2wb9p\" (UniqueName: \"kubernetes.io/projected/5371c07a-6563-4db1-9483-32e9300a0581-kube-api-access-2wb9p\") pod \"5371c07a-6563-4db1-9483-32e9300a0581\" (UID: \"5371c07a-6563-4db1-9483-32e9300a0581\") " Oct 30 01:35:03.149554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:03.145303 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5371c07a-6563-4db1-9483-32e9300a0581-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5371c07a-6563-4db1-9483-32e9300a0581-openshift-service-ca\") pod \"5371c07a-6563-4db1-9483-32e9300a0581\" (UID: \"5371c07a-6563-4db1-9483-32e9300a0581\") " Oct 30 01:35:03.149554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:03.145506 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5371c07a-6563-4db1-9483-32e9300a0581-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5371c07a-6563-4db1-9483-32e9300a0581-maas-api-tls\") pod \"5371c07a-6563-4db1-9483-32e9300a0581\" (UID: \"5371c07a-6563-4db1-9483-32e9300a0581\") " Oct 30 01:35:03.167554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:03.165901 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5371c07a-6563-4db1-9483-32e9300a0581-openshift-service-ca" pod "5371c07a-6563-4db1-9483-32e9300a0581" (UID: "5371c07a-6563-4db1-9483-32e9300a0581"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:35:03.191554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:03.188760 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5371c07a-6563-4db1-9483-32e9300a0581-kube-api-access-2wb9p" pod "5371c07a-6563-4db1-9483-32e9300a0581" (UID: "5371c07a-6563-4db1-9483-32e9300a0581"). InnerVolumeSpecName "kube-api-access-2wb9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:35:03.191554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:03.188867 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5371c07a-6563-4db1-9483-32e9300a0581-metrics-certs" pod "5371c07a-6563-4db1-9483-32e9300a0581" (UID: "5371c07a-6563-4db1-9483-32e9300a0581"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:35:03.191554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:03.188919 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5371c07a-6563-4db1-9483-32e9300a0581-maas-api-tls" pod "5371c07a-6563-4db1-9483-32e9300a0581" (UID: "5371c07a-6563-4db1-9483-32e9300a0581"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:35:03.253606 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:03.249113 6221 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5371c07a-6563-4db1-9483-32e9300a0581-maas-api-tls\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:35:03.253606 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:03.249150 6221 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5371c07a-6563-4db1-9483-32e9300a0581-metrics-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:35:03.253606 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:03.249165 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wb9p\" (UniqueName: \"kubernetes.io/projected/5371c07a-6563-4db1-9483-32e9300a0581-kube-api-access-2wb9p\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:35:03.253606 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:03.249179 6221 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5371c07a-6563-4db1-9483-32e9300a0581-openshift-service-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:35:03.763557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:03.750158 6221 generic.go:356] "Generic (PLEG): container finished" podID="5371c07a-6563-4db1-9483-32e9300a0581" containerID="7a749af26f034e0988a412a349f35edba89574946bb69cfe7380931e74f63330" exitCode=0 Oct 30 01:35:03.763557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:03.755848 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-x4w45" Oct 30 01:35:03.796560 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:03.793687 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-x4w45" event={"ID":"5371c07a-6563-4db1-9483-32e9300a0581","Type":"ContainerDied","Data":"7a749af26f034e0988a412a349f35edba89574946bb69cfe7380931e74f63330"} Oct 30 01:35:03.796560 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:03.793787 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-x4w45" event={"ID":"5371c07a-6563-4db1-9483-32e9300a0581","Type":"ContainerDied","Data":"4473e51f9457561c7050252b2f025adabfd69ec94163137e508ae836e4b3de3a"} Oct 30 01:35:03.796560 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:03.793827 6221 scope.go:122] "RemoveContainer" containerID="7a749af26f034e0988a412a349f35edba89574946bb69cfe7380931e74f63330" Oct 30 01:35:03.875214 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:03.875181 6221 scope.go:122] "RemoveContainer" containerID="7a749af26f034e0988a412a349f35edba89574946bb69cfe7380931e74f63330" Oct 30 01:35:03.879805 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:35:03.879753 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a749af26f034e0988a412a349f35edba89574946bb69cfe7380931e74f63330\": container with ID starting with 7a749af26f034e0988a412a349f35edba89574946bb69cfe7380931e74f63330 not found: ID does not exist" containerID="7a749af26f034e0988a412a349f35edba89574946bb69cfe7380931e74f63330" Oct 30 01:35:03.880057 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:03.880022 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a749af26f034e0988a412a349f35edba89574946bb69cfe7380931e74f63330"} err="failed to get container status \"7a749af26f034e0988a412a349f35edba89574946bb69cfe7380931e74f63330\": rpc error: code = NotFound desc = could not find container \"7a749af26f034e0988a412a349f35edba89574946bb69cfe7380931e74f63330\": container with ID starting with 7a749af26f034e0988a412a349f35edba89574946bb69cfe7380931e74f63330 not found: ID does not exist" Oct 30 01:35:03.901831 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:03.901794 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-x4w45"] Oct 30 01:35:03.914363 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:03.914288 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-278fe7d5-59f4cbfd44-x4w45"] Oct 30 01:35:04.012094 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:04.012052 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-wxkcd" podUID="41211dad-aa70-47ff-b6bb-12426581ab00" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:35:04.548051 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:04.548015 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5371c07a-6563-4db1-9483-32e9300a0581" path="/var/lib/kubelet/pods/5371c07a-6563-4db1-9483-32e9300a0581/volumes" Oct 30 01:35:04.683539 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:04.683504 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-97c137-64d87865f-d68pz"] Oct 30 01:35:04.684208 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:04.684190 6221 cpu_manager.go:414] "containerMap: removing container" podUID="5371c07a-6563-4db1-9483-32e9300a0581" containerName="maas-api" Oct 30 01:35:04.684254 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:04.684210 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5371c07a-6563-4db1-9483-32e9300a0581" containerName="maas-api" Oct 30 01:35:04.684324 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:04.684313 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="5371c07a-6563-4db1-9483-32e9300a0581" containerName="maas-api" Oct 30 01:35:04.687520 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:04.687499 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-97c137-64d87865f-d68pz" Oct 30 01:35:04.705136 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:04.705104 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-97c137-64d87865f-d68pz"] Oct 30 01:35:04.705423 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:04.705391 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-97c137\"" Oct 30 01:35:04.705621 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:04.705604 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-97c137-dockercfg-jt266\"" Oct 30 01:35:04.880613 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:04.824499 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d280af82-ee8f-4f27-8cbb-2e7fbd80b5d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-97c137-64d87865f-d68pz\" (UID: \"d280af82-ee8f-4f27-8cbb-2e7fbd80b5d1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-97c137-64d87865f-d68pz" Oct 30 01:35:04.880613 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:04.858864 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48rz8\" (UniqueName: \"kubernetes.io/projected/d280af82-ee8f-4f27-8cbb-2e7fbd80b5d1-kube-api-access-48rz8\") pod \"payload-pre-processing-e2e-api-b-97c137-64d87865f-d68pz\" (UID: \"d280af82-ee8f-4f27-8cbb-2e7fbd80b5d1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-97c137-64d87865f-d68pz" Oct 30 01:35:04.971556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:04.971137 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48rz8\" (UniqueName: \"kubernetes.io/projected/d280af82-ee8f-4f27-8cbb-2e7fbd80b5d1-kube-api-access-48rz8\") pod \"payload-pre-processing-e2e-api-b-97c137-64d87865f-d68pz\" (UID: \"d280af82-ee8f-4f27-8cbb-2e7fbd80b5d1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-97c137-64d87865f-d68pz" Oct 30 01:35:04.971556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:04.971386 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d280af82-ee8f-4f27-8cbb-2e7fbd80b5d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-97c137-64d87865f-d68pz\" (UID: \"d280af82-ee8f-4f27-8cbb-2e7fbd80b5d1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-97c137-64d87865f-d68pz" Oct 30 01:35:04.982071 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:04.972740 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d280af82-ee8f-4f27-8cbb-2e7fbd80b5d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-97c137-64d87865f-d68pz\" (UID: \"d280af82-ee8f-4f27-8cbb-2e7fbd80b5d1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-97c137-64d87865f-d68pz" Oct 30 01:35:05.022025 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:05.021953 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-97c137-7c4fbd7b9b-rh7wf"] Oct 30 01:35:05.088392 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:05.088364 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48rz8\" (UniqueName: \"kubernetes.io/projected/d280af82-ee8f-4f27-8cbb-2e7fbd80b5d1-kube-api-access-48rz8\") pod \"payload-pre-processing-e2e-api-b-97c137-64d87865f-d68pz\" (UID: \"d280af82-ee8f-4f27-8cbb-2e7fbd80b5d1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-97c137-64d87865f-d68pz" Oct 30 01:35:05.096419 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:05.092743 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-97c137-7c4fbd7b9b-rh7wf"] Oct 30 01:35:05.096655 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:05.092840 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-97c137-7c4fbd7b9b-rh7wf" Oct 30 01:35:05.197832 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:05.197737 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8hd4\" (UniqueName: \"kubernetes.io/projected/6d161227-4d8e-49b1-b4e3-308d91f85588-kube-api-access-p8hd4\") pod \"payload-processing-e2e-api-b-97c137-7c4fbd7b9b-rh7wf\" (UID: \"6d161227-4d8e-49b1-b4e3-308d91f85588\") " pod="openshift-ingress/payload-processing-e2e-api-b-97c137-7c4fbd7b9b-rh7wf" Oct 30 01:35:05.198020 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:05.197924 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d161227-4d8e-49b1-b4e3-308d91f85588-plugins-config-volume\") pod \"payload-processing-e2e-api-b-97c137-7c4fbd7b9b-rh7wf\" (UID: \"6d161227-4d8e-49b1-b4e3-308d91f85588\") " pod="openshift-ingress/payload-processing-e2e-api-b-97c137-7c4fbd7b9b-rh7wf" Oct 30 01:35:05.299826 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:05.299786 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d161227-4d8e-49b1-b4e3-308d91f85588-plugins-config-volume\") pod \"payload-processing-e2e-api-b-97c137-7c4fbd7b9b-rh7wf\" (UID: \"6d161227-4d8e-49b1-b4e3-308d91f85588\") " pod="openshift-ingress/payload-processing-e2e-api-b-97c137-7c4fbd7b9b-rh7wf" Oct 30 01:35:05.299979 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:05.299965 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8hd4\" (UniqueName: \"kubernetes.io/projected/6d161227-4d8e-49b1-b4e3-308d91f85588-kube-api-access-p8hd4\") pod \"payload-processing-e2e-api-b-97c137-7c4fbd7b9b-rh7wf\" (UID: \"6d161227-4d8e-49b1-b4e3-308d91f85588\") " pod="openshift-ingress/payload-processing-e2e-api-b-97c137-7c4fbd7b9b-rh7wf" Oct 30 01:35:05.300600 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:05.300579 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d161227-4d8e-49b1-b4e3-308d91f85588-plugins-config-volume\") pod \"payload-processing-e2e-api-b-97c137-7c4fbd7b9b-rh7wf\" (UID: \"6d161227-4d8e-49b1-b4e3-308d91f85588\") " pod="openshift-ingress/payload-processing-e2e-api-b-97c137-7c4fbd7b9b-rh7wf" Oct 30 01:35:05.301261 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:05.301242 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-97c137-64d87865f-d68pz" Oct 30 01:35:05.319501 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:05.319476 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8hd4\" (UniqueName: \"kubernetes.io/projected/6d161227-4d8e-49b1-b4e3-308d91f85588-kube-api-access-p8hd4\") pod \"payload-processing-e2e-api-b-97c137-7c4fbd7b9b-rh7wf\" (UID: \"6d161227-4d8e-49b1-b4e3-308d91f85588\") " pod="openshift-ingress/payload-processing-e2e-api-b-97c137-7c4fbd7b9b-rh7wf" Oct 30 01:35:05.416839 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:05.416804 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-97c137-7c4fbd7b9b-rh7wf" Oct 30 01:35:05.610010 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:05.609985 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-97c137-64d87865f-d68pz"] Oct 30 01:35:05.612117 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:35:05.612085 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd280af82_ee8f_4f27_8cbb_2e7fbd80b5d1.slice/crio-399d48f43fba5ef1ea2888f8df119d9226abdafa1e322fe626586be34f881eb4 WatchSource:0}: Error finding container 399d48f43fba5ef1ea2888f8df119d9226abdafa1e322fe626586be34f881eb4: Status 404 returned error can't find the container with id 399d48f43fba5ef1ea2888f8df119d9226abdafa1e322fe626586be34f881eb4 Oct 30 01:35:05.739416 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:05.739364 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-97c137-7c4fbd7b9b-rh7wf"] Oct 30 01:35:05.741498 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:35:05.741472 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d161227_4d8e_49b1_b4e3_308d91f85588.slice/crio-ef36a3530f47557730a95962035308133d83530667e6f9a11510828901749405 WatchSource:0}: Error finding container ef36a3530f47557730a95962035308133d83530667e6f9a11510828901749405: Status 404 returned error can't find the container with id ef36a3530f47557730a95962035308133d83530667e6f9a11510828901749405 Oct 30 01:35:06.103999 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:06.103957 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-97c137-64d87865f-d68pz" event={"ID":"d280af82-ee8f-4f27-8cbb-2e7fbd80b5d1","Type":"ContainerStarted","Data":"4966adbefe210ada5c23be855118b62f1ed74d079638bb32ff3079dad5b01c75"} Oct 30 01:35:06.104178 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:06.104025 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-97c137-64d87865f-d68pz" event={"ID":"d280af82-ee8f-4f27-8cbb-2e7fbd80b5d1","Type":"ContainerStarted","Data":"399d48f43fba5ef1ea2888f8df119d9226abdafa1e322fe626586be34f881eb4"} Oct 30 01:35:06.104178 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:06.104090 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-97c137-64d87865f-d68pz" Oct 30 01:35:06.106281 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:06.106252 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-97c137-7c4fbd7b9b-rh7wf" event={"ID":"6d161227-4d8e-49b1-b4e3-308d91f85588","Type":"ContainerStarted","Data":"3faebd6e277c3480b4230051f4cbf49d8a48b1cb42fb4975b51ecf7af0f6fc9d"} Oct 30 01:35:06.106420 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:06.106297 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-97c137-7c4fbd7b9b-rh7wf" event={"ID":"6d161227-4d8e-49b1-b4e3-308d91f85588","Type":"ContainerStarted","Data":"ef36a3530f47557730a95962035308133d83530667e6f9a11510828901749405"} Oct 30 01:35:06.106420 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:06.106317 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-97c137-7c4fbd7b9b-rh7wf" Oct 30 01:35:06.147471 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:06.147406 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-97c137-64d87865f-d68pz" podStartSLOduration=2.14738495 podStartE2EDuration="2.14738495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:35:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:35:06.144388446 +0000 UTC m=+1029.722675782" watchObservedRunningTime="2026-10-30 01:35:06.14738495 +0000 UTC m=+1029.725672287" Oct 30 01:35:06.184828 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:06.184773 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-97c137-7c4fbd7b9b-rh7wf" podStartSLOduration=2.184755971 podStartE2EDuration="2.184755971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:35:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:35:06.182747132 +0000 UTC m=+1029.761034476" watchObservedRunningTime="2026-10-30 01:35:06.184755971 +0000 UTC m=+1029.763043306" Oct 30 01:35:06.413959 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:06.413878 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-288c8f-5db9bddc67-hz2dc"] Oct 30 01:35:06.417522 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:06.417497 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-288c8f-5db9bddc67-hz2dc" Oct 30 01:35:06.435089 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:06.435057 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-288c8f\"" Oct 30 01:35:06.436802 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:06.436766 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-288c8f-5db9bddc67-hz2dc"] Oct 30 01:35:06.536710 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:06.532405 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c793208-af28-4c73-a6de-9ea3f69e80d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-288c8f-5db9bddc67-hz2dc\" (UID: \"0c793208-af28-4c73-a6de-9ea3f69e80d6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-288c8f-5db9bddc67-hz2dc" Oct 30 01:35:06.536710 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:06.532613 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxnnv\" (UniqueName: \"kubernetes.io/projected/0c793208-af28-4c73-a6de-9ea3f69e80d6-kube-api-access-nxnnv\") pod \"payload-pre-processing-e2e-api-a-288c8f-5db9bddc67-hz2dc\" (UID: \"0c793208-af28-4c73-a6de-9ea3f69e80d6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-288c8f-5db9bddc67-hz2dc" Oct 30 01:35:06.664615 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:06.649861 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c793208-af28-4c73-a6de-9ea3f69e80d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-288c8f-5db9bddc67-hz2dc\" (UID: \"0c793208-af28-4c73-a6de-9ea3f69e80d6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-288c8f-5db9bddc67-hz2dc" Oct 30 01:35:06.664615 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:06.650048 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxnnv\" (UniqueName: \"kubernetes.io/projected/0c793208-af28-4c73-a6de-9ea3f69e80d6-kube-api-access-nxnnv\") pod \"payload-pre-processing-e2e-api-a-288c8f-5db9bddc67-hz2dc\" (UID: \"0c793208-af28-4c73-a6de-9ea3f69e80d6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-288c8f-5db9bddc67-hz2dc" Oct 30 01:35:06.664615 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:06.651615 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c793208-af28-4c73-a6de-9ea3f69e80d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-288c8f-5db9bddc67-hz2dc\" (UID: \"0c793208-af28-4c73-a6de-9ea3f69e80d6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-288c8f-5db9bddc67-hz2dc" Oct 30 01:35:06.768560 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:06.752953 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxnnv\" (UniqueName: \"kubernetes.io/projected/0c793208-af28-4c73-a6de-9ea3f69e80d6-kube-api-access-nxnnv\") pod \"payload-pre-processing-e2e-api-a-288c8f-5db9bddc67-hz2dc\" (UID: \"0c793208-af28-4c73-a6de-9ea3f69e80d6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-288c8f-5db9bddc67-hz2dc" Oct 30 01:35:06.768560 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:06.754560 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-288c8f-5db9bddc67-hz2dc" Oct 30 01:35:07.060402 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:07.060376 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-wxkcd" Oct 30 01:35:07.115673 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:07.115639 6221 generic.go:356] "Generic (PLEG): container finished" podID="41211dad-aa70-47ff-b6bb-12426581ab00" containerID="465226a307196f73a364d6d233cfd00a766f70de1bcf8c822661103ef0647471" exitCode=0 Oct 30 01:35:07.115866 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:07.115694 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-wxkcd" Oct 30 01:35:07.115866 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:07.115790 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-wxkcd" event={"ID":"41211dad-aa70-47ff-b6bb-12426581ab00","Type":"ContainerDied","Data":"465226a307196f73a364d6d233cfd00a766f70de1bcf8c822661103ef0647471"} Oct 30 01:35:07.115866 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:07.115838 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-wxkcd" event={"ID":"41211dad-aa70-47ff-b6bb-12426581ab00","Type":"ContainerDied","Data":"279351eb413defaeab39a411443fe4e6f8045d218e588660d12c26dec30e5045"} Oct 30 01:35:07.115866 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:07.115866 6221 scope.go:122] "RemoveContainer" containerID="465226a307196f73a364d6d233cfd00a766f70de1bcf8c822661103ef0647471" Oct 30 01:35:07.127027 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:07.127000 6221 scope.go:122] "RemoveContainer" containerID="465226a307196f73a364d6d233cfd00a766f70de1bcf8c822661103ef0647471" Oct 30 01:35:07.127371 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:35:07.127350 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"465226a307196f73a364d6d233cfd00a766f70de1bcf8c822661103ef0647471\": container with ID starting with 465226a307196f73a364d6d233cfd00a766f70de1bcf8c822661103ef0647471 not found: ID does not exist" containerID="465226a307196f73a364d6d233cfd00a766f70de1bcf8c822661103ef0647471" Oct 30 01:35:07.127426 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:07.127382 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"465226a307196f73a364d6d233cfd00a766f70de1bcf8c822661103ef0647471"} err="failed to get container status \"465226a307196f73a364d6d233cfd00a766f70de1bcf8c822661103ef0647471\": rpc error: code = NotFound desc = could not find container \"465226a307196f73a364d6d233cfd00a766f70de1bcf8c822661103ef0647471\": container with ID starting with 465226a307196f73a364d6d233cfd00a766f70de1bcf8c822661103ef0647471 not found: ID does not exist" Oct 30 01:35:07.150450 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:07.150419 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-288c8f-5db9bddc67-hz2dc"] Oct 30 01:35:07.159572 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:35:07.152930 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c793208_af28_4c73_a6de_9ea3f69e80d6.slice/crio-87fb890724504bd9045e7629f18009d8f42e5dcd9be1a3617c3cc8345458027d WatchSource:0}: Error finding container 87fb890724504bd9045e7629f18009d8f42e5dcd9be1a3617c3cc8345458027d: Status 404 returned error can't find the container with id 87fb890724504bd9045e7629f18009d8f42e5dcd9be1a3617c3cc8345458027d Oct 30 01:35:07.159572 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:07.157082 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41211dad-aa70-47ff-b6bb-12426581ab00-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41211dad-aa70-47ff-b6bb-12426581ab00-plugins-config-volume\") pod \"41211dad-aa70-47ff-b6bb-12426581ab00\" (UID: \"41211dad-aa70-47ff-b6bb-12426581ab00\") " Oct 30 01:35:07.159572 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:07.157254 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41211dad-aa70-47ff-b6bb-12426581ab00-kube-api-access-8c5xx\" (UniqueName: \"kubernetes.io/projected/41211dad-aa70-47ff-b6bb-12426581ab00-kube-api-access-8c5xx\") pod \"41211dad-aa70-47ff-b6bb-12426581ab00\" (UID: \"41211dad-aa70-47ff-b6bb-12426581ab00\") " Oct 30 01:35:07.159572 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:07.157397 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41211dad-aa70-47ff-b6bb-12426581ab00-plugins-config-volume" pod "41211dad-aa70-47ff-b6bb-12426581ab00" (UID: "41211dad-aa70-47ff-b6bb-12426581ab00"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:35:07.159572 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:07.157878 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41211dad-aa70-47ff-b6bb-12426581ab00-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:35:07.160238 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:07.159691 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41211dad-aa70-47ff-b6bb-12426581ab00-kube-api-access-8c5xx" pod "41211dad-aa70-47ff-b6bb-12426581ab00" (UID: "41211dad-aa70-47ff-b6bb-12426581ab00"). InnerVolumeSpecName "kube-api-access-8c5xx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:35:07.259460 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:07.259426 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c5xx\" (UniqueName: \"kubernetes.io/projected/41211dad-aa70-47ff-b6bb-12426581ab00-kube-api-access-8c5xx\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:35:07.486558 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:07.486449 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-wxkcd"] Oct 30 01:35:07.497283 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:07.497230 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-wxkcd"] Oct 30 01:35:08.125741 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:08.125702 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-288c8f-5db9bddc67-hz2dc" event={"ID":"0c793208-af28-4c73-a6de-9ea3f69e80d6","Type":"ContainerStarted","Data":"0728577adfc60d8fe122811fd71ceb0736812f1928ccf48c99877f4f728b0e3e"} Oct 30 01:35:08.126181 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:08.125802 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-288c8f-5db9bddc67-hz2dc" event={"ID":"0c793208-af28-4c73-a6de-9ea3f69e80d6","Type":"ContainerStarted","Data":"87fb890724504bd9045e7629f18009d8f42e5dcd9be1a3617c3cc8345458027d"} Oct 30 01:35:08.126181 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:08.125937 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-288c8f-5db9bddc67-hz2dc" Oct 30 01:35:08.166996 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:08.166935 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-288c8f-5db9bddc67-hz2dc" podStartSLOduration=2.166919244 podStartE2EDuration="2.166919244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:35:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:35:08.163909381 +0000 UTC m=+1031.742196724" watchObservedRunningTime="2026-10-30 01:35:08.166919244 +0000 UTC m=+1031.745206582" Oct 30 01:35:08.548726 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:08.548693 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41211dad-aa70-47ff-b6bb-12426581ab00" path="/var/lib/kubelet/pods/41211dad-aa70-47ff-b6bb-12426581ab00/volumes" Oct 30 01:35:09.062312 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:09.062279 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:35:16.773520 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:16.773486 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-qd775"] Oct 30 01:35:16.774159 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:16.774146 6221 cpu_manager.go:414] "containerMap: removing container" podUID="41211dad-aa70-47ff-b6bb-12426581ab00" containerName="payload-pre-processing" Oct 30 01:35:16.774201 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:16.774161 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41211dad-aa70-47ff-b6bb-12426581ab00" containerName="payload-pre-processing" Oct 30 01:35:16.774273 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:16.774264 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="41211dad-aa70-47ff-b6bb-12426581ab00" containerName="payload-pre-processing" Oct 30 01:35:16.777891 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:16.777871 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-qd775" Oct 30 01:35:16.794481 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:16.794458 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-qd775"] Oct 30 01:35:16.802694 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:16.802665 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-278fe7d5\"" Oct 30 01:35:16.802694 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:16.802666 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-278fe7d5-dockercfg-rdtnj\"" Oct 30 01:35:16.804459 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:16.804435 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ce0d4a4-6037-4aff-b762-f9ed9c0f1816-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-qd775\" (UID: \"0ce0d4a4-6037-4aff-b762-f9ed9c0f1816\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-qd775" Oct 30 01:35:16.804597 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:16.804501 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9smz8\" (UniqueName: \"kubernetes.io/projected/0ce0d4a4-6037-4aff-b762-f9ed9c0f1816-kube-api-access-9smz8\") pod \"payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-qd775\" (UID: \"0ce0d4a4-6037-4aff-b762-f9ed9c0f1816\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-qd775" Oct 30 01:35:16.920570 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:16.915450 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ce0d4a4-6037-4aff-b762-f9ed9c0f1816-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-qd775\" (UID: \"0ce0d4a4-6037-4aff-b762-f9ed9c0f1816\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-qd775" Oct 30 01:35:16.920570 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:16.915574 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9smz8\" (UniqueName: \"kubernetes.io/projected/0ce0d4a4-6037-4aff-b762-f9ed9c0f1816-kube-api-access-9smz8\") pod \"payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-qd775\" (UID: \"0ce0d4a4-6037-4aff-b762-f9ed9c0f1816\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-qd775" Oct 30 01:35:16.920570 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:16.918982 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ce0d4a4-6037-4aff-b762-f9ed9c0f1816-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-qd775\" (UID: \"0ce0d4a4-6037-4aff-b762-f9ed9c0f1816\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-qd775" Oct 30 01:35:17.141551 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:17.140063 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-97c137-64d87865f-d68pz" Oct 30 01:35:17.151347 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:17.151312 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-97c137-7c4fbd7b9b-rh7wf" Oct 30 01:35:17.163905 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:17.163872 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9smz8\" (UniqueName: \"kubernetes.io/projected/0ce0d4a4-6037-4aff-b762-f9ed9c0f1816-kube-api-access-9smz8\") pod \"payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-qd775\" (UID: \"0ce0d4a4-6037-4aff-b762-f9ed9c0f1816\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-qd775" Oct 30 01:35:17.393073 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:17.392996 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-qd775" Oct 30 01:35:17.709012 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:17.708984 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-qd775"] Oct 30 01:35:17.711214 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:35:17.711190 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ce0d4a4_6037_4aff_b762_f9ed9c0f1816.slice/crio-1f02c93aabcba62f0ce2ae772b0b26c347a4fd3e2244f1718c124492580d6bc3 WatchSource:0}: Error finding container 1f02c93aabcba62f0ce2ae772b0b26c347a4fd3e2244f1718c124492580d6bc3: Status 404 returned error can't find the container with id 1f02c93aabcba62f0ce2ae772b0b26c347a4fd3e2244f1718c124492580d6bc3 Oct 30 01:35:18.299398 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:18.299349 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-qd775" event={"ID":"0ce0d4a4-6037-4aff-b762-f9ed9c0f1816","Type":"ContainerStarted","Data":"32a7d9570e7b6cc0537c7cac808e00eee56b015f3cbd269caacad84198b6eca6"} Oct 30 01:35:18.299919 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:18.299479 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-qd775" event={"ID":"0ce0d4a4-6037-4aff-b762-f9ed9c0f1816","Type":"ContainerStarted","Data":"1f02c93aabcba62f0ce2ae772b0b26c347a4fd3e2244f1718c124492580d6bc3"} Oct 30 01:35:18.300619 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:18.300594 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-qd775" Oct 30 01:35:18.404552 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:18.396495 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-qd775" podStartSLOduration=2.396471407 podStartE2EDuration="2.396471407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:35:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:35:18.357048501 +0000 UTC m=+1041.935335839" watchObservedRunningTime="2026-10-30 01:35:18.396471407 +0000 UTC m=+1041.974758752" Oct 30 01:35:19.135609 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:19.135583 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-288c8f-5db9bddc67-hz2dc" Oct 30 01:35:21.494672 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:35:21.494632 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:35:30.387389 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:30.387354 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-qd775" Oct 30 01:35:34.678434 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:34.678394 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4d38e5-569dc98886-flmzc"] Oct 30 01:35:34.709770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:34.709742 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4d38e5-775c67c59-wrshf"] Oct 30 01:35:39.684558 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:39.681965 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-4d38e5-569dc98886-flmzc" podUID="bffcc9b8-9193-4110-950c-fc16323075e8" containerName="payload-processing" containerID="cri-o://6945d0238df51b9d8c253f006d0ca33af12b469359684574d489c4972ad1a832" gracePeriod=25 Oct 30 01:35:39.714553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:39.710602 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4d38e5-775c67c59-wrshf" podUID="32b0bea5-05bb-4c7a-942d-eb057e672324" containerName="payload-pre-processing" containerID="cri-o://5e130e589f41ef91d95ba72f7874e9e1a20b651c58e0b7a5eac7d98198d062bf" gracePeriod=25 Oct 30 01:35:40.263609 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:40.262840 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4d38e5-569dc98886-flmzc" Oct 30 01:35:40.288468 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:40.288443 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4d38e5-775c67c59-wrshf" Oct 30 01:35:40.378981 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:40.378938 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32b0bea5-05bb-4c7a-942d-eb057e672324-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32b0bea5-05bb-4c7a-942d-eb057e672324-plugins-config-volume\") pod \"32b0bea5-05bb-4c7a-942d-eb057e672324\" (UID: \"32b0bea5-05bb-4c7a-942d-eb057e672324\") " Oct 30 01:35:40.379186 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:40.379170 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bffcc9b8-9193-4110-950c-fc16323075e8-kube-api-access-wfl9x\" (UniqueName: \"kubernetes.io/projected/bffcc9b8-9193-4110-950c-fc16323075e8-kube-api-access-wfl9x\") pod \"bffcc9b8-9193-4110-950c-fc16323075e8\" (UID: \"bffcc9b8-9193-4110-950c-fc16323075e8\") " Oct 30 01:35:40.379263 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:40.379196 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32b0bea5-05bb-4c7a-942d-eb057e672324-kube-api-access-z4mbc\" (UniqueName: \"kubernetes.io/projected/32b0bea5-05bb-4c7a-942d-eb057e672324-kube-api-access-z4mbc\") pod \"32b0bea5-05bb-4c7a-942d-eb057e672324\" (UID: \"32b0bea5-05bb-4c7a-942d-eb057e672324\") " Oct 30 01:35:40.379393 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:40.379365 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32b0bea5-05bb-4c7a-942d-eb057e672324-plugins-config-volume" pod "32b0bea5-05bb-4c7a-942d-eb057e672324" (UID: "32b0bea5-05bb-4c7a-942d-eb057e672324"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:35:40.379744 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:40.379722 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bffcc9b8-9193-4110-950c-fc16323075e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bffcc9b8-9193-4110-950c-fc16323075e8-plugins-config-volume\") pod \"bffcc9b8-9193-4110-950c-fc16323075e8\" (UID: \"bffcc9b8-9193-4110-950c-fc16323075e8\") " Oct 30 01:35:40.380981 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:40.380336 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bffcc9b8-9193-4110-950c-fc16323075e8-plugins-config-volume" pod "bffcc9b8-9193-4110-950c-fc16323075e8" (UID: "bffcc9b8-9193-4110-950c-fc16323075e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:35:40.381105 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:40.381050 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bffcc9b8-9193-4110-950c-fc16323075e8-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:35:40.381105 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:40.381075 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/32b0bea5-05bb-4c7a-942d-eb057e672324-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:35:40.381595 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:40.381570 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bffcc9b8-9193-4110-950c-fc16323075e8-kube-api-access-wfl9x" pod "bffcc9b8-9193-4110-950c-fc16323075e8" (UID: "bffcc9b8-9193-4110-950c-fc16323075e8"). InnerVolumeSpecName "kube-api-access-wfl9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:35:40.381775 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:40.381754 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32b0bea5-05bb-4c7a-942d-eb057e672324-kube-api-access-z4mbc" pod "32b0bea5-05bb-4c7a-942d-eb057e672324" (UID: "32b0bea5-05bb-4c7a-942d-eb057e672324"). InnerVolumeSpecName "kube-api-access-z4mbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:35:40.483601 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:40.483564 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfl9x\" (UniqueName: \"kubernetes.io/projected/bffcc9b8-9193-4110-950c-fc16323075e8-kube-api-access-wfl9x\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:35:40.483601 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:40.483593 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4mbc\" (UniqueName: \"kubernetes.io/projected/32b0bea5-05bb-4c7a-942d-eb057e672324-kube-api-access-z4mbc\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:35:40.663892 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:40.663855 6221 generic.go:356] "Generic (PLEG): container finished" podID="bffcc9b8-9193-4110-950c-fc16323075e8" containerID="6945d0238df51b9d8c253f006d0ca33af12b469359684574d489c4972ad1a832" exitCode=0 Oct 30 01:35:40.664091 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:40.663916 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4d38e5-569dc98886-flmzc" Oct 30 01:35:40.664091 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:40.663983 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4d38e5-569dc98886-flmzc" event={"ID":"bffcc9b8-9193-4110-950c-fc16323075e8","Type":"ContainerDied","Data":"6945d0238df51b9d8c253f006d0ca33af12b469359684574d489c4972ad1a832"} Oct 30 01:35:40.664091 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:40.664040 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4d38e5-569dc98886-flmzc" event={"ID":"bffcc9b8-9193-4110-950c-fc16323075e8","Type":"ContainerDied","Data":"42b72d2ca8d2e819005d73f0385266730416860d0e3b53e2717e5f6d7e7aa01d"} Oct 30 01:35:40.664091 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:40.664070 6221 scope.go:122] "RemoveContainer" containerID="6945d0238df51b9d8c253f006d0ca33af12b469359684574d489c4972ad1a832" Oct 30 01:35:40.666559 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:40.666518 6221 generic.go:356] "Generic (PLEG): container finished" podID="32b0bea5-05bb-4c7a-942d-eb057e672324" containerID="5e130e589f41ef91d95ba72f7874e9e1a20b651c58e0b7a5eac7d98198d062bf" exitCode=0 Oct 30 01:35:40.666681 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:40.666577 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4d38e5-775c67c59-wrshf" event={"ID":"32b0bea5-05bb-4c7a-942d-eb057e672324","Type":"ContainerDied","Data":"5e130e589f41ef91d95ba72f7874e9e1a20b651c58e0b7a5eac7d98198d062bf"} Oct 30 01:35:40.666681 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:40.666596 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4d38e5-775c67c59-wrshf" Oct 30 01:35:40.666681 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:40.666616 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4d38e5-775c67c59-wrshf" event={"ID":"32b0bea5-05bb-4c7a-942d-eb057e672324","Type":"ContainerDied","Data":"0b8e167908fa1eeb0fb095a47f96f2f7b504651b22b878d50332e9b0b06dce0b"} Oct 30 01:35:40.676142 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:40.676120 6221 scope.go:122] "RemoveContainer" containerID="6945d0238df51b9d8c253f006d0ca33af12b469359684574d489c4972ad1a832" Oct 30 01:35:40.676476 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:35:40.676462 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6945d0238df51b9d8c253f006d0ca33af12b469359684574d489c4972ad1a832\": container with ID starting with 6945d0238df51b9d8c253f006d0ca33af12b469359684574d489c4972ad1a832 not found: ID does not exist" containerID="6945d0238df51b9d8c253f006d0ca33af12b469359684574d489c4972ad1a832" Oct 30 01:35:40.676609 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:40.676591 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6945d0238df51b9d8c253f006d0ca33af12b469359684574d489c4972ad1a832"} err="failed to get container status \"6945d0238df51b9d8c253f006d0ca33af12b469359684574d489c4972ad1a832\": rpc error: code = NotFound desc = could not find container \"6945d0238df51b9d8c253f006d0ca33af12b469359684574d489c4972ad1a832\": container with ID starting with 6945d0238df51b9d8c253f006d0ca33af12b469359684574d489c4972ad1a832 not found: ID does not exist" Oct 30 01:35:40.676704 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:40.676695 6221 scope.go:122] "RemoveContainer" containerID="5e130e589f41ef91d95ba72f7874e9e1a20b651c58e0b7a5eac7d98198d062bf" Oct 30 01:35:40.688890 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:40.688645 6221 scope.go:122] "RemoveContainer" containerID="5e130e589f41ef91d95ba72f7874e9e1a20b651c58e0b7a5eac7d98198d062bf" Oct 30 01:35:40.689200 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:35:40.689009 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e130e589f41ef91d95ba72f7874e9e1a20b651c58e0b7a5eac7d98198d062bf\": container with ID starting with 5e130e589f41ef91d95ba72f7874e9e1a20b651c58e0b7a5eac7d98198d062bf not found: ID does not exist" containerID="5e130e589f41ef91d95ba72f7874e9e1a20b651c58e0b7a5eac7d98198d062bf" Oct 30 01:35:40.689200 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:40.689040 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e130e589f41ef91d95ba72f7874e9e1a20b651c58e0b7a5eac7d98198d062bf"} err="failed to get container status \"5e130e589f41ef91d95ba72f7874e9e1a20b651c58e0b7a5eac7d98198d062bf\": rpc error: code = NotFound desc = could not find container \"5e130e589f41ef91d95ba72f7874e9e1a20b651c58e0b7a5eac7d98198d062bf\": container with ID starting with 5e130e589f41ef91d95ba72f7874e9e1a20b651c58e0b7a5eac7d98198d062bf not found: ID does not exist" Oct 30 01:35:40.767118 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:40.766963 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4d38e5-569dc98886-flmzc"] Oct 30 01:35:40.770662 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:40.770633 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4d38e5-569dc98886-flmzc"] Oct 30 01:35:40.803060 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:40.803014 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4d38e5-775c67c59-wrshf"] Oct 30 01:35:40.831271 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:40.831176 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4d38e5-775c67c59-wrshf"] Oct 30 01:35:42.550639 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:42.550605 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32b0bea5-05bb-4c7a-942d-eb057e672324" path="/var/lib/kubelet/pods/32b0bea5-05bb-4c7a-942d-eb057e672324/volumes" Oct 30 01:35:42.551255 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:42.551234 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bffcc9b8-9193-4110-950c-fc16323075e8" path="/var/lib/kubelet/pods/bffcc9b8-9193-4110-950c-fc16323075e8/volumes" Oct 30 01:35:44.921766 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:44.921723 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg"] Oct 30 01:35:44.922635 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:44.922613 6221 cpu_manager.go:414] "containerMap: removing container" podUID="bffcc9b8-9193-4110-950c-fc16323075e8" containerName="payload-processing" Oct 30 01:35:44.922635 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:44.922635 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bffcc9b8-9193-4110-950c-fc16323075e8" containerName="payload-processing" Oct 30 01:35:44.922796 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:44.922661 6221 cpu_manager.go:414] "containerMap: removing container" podUID="32b0bea5-05bb-4c7a-942d-eb057e672324" containerName="payload-pre-processing" Oct 30 01:35:44.922796 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:44.922671 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32b0bea5-05bb-4c7a-942d-eb057e672324" containerName="payload-pre-processing" Oct 30 01:35:44.922901 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:44.922835 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="bffcc9b8-9193-4110-950c-fc16323075e8" containerName="payload-processing" Oct 30 01:35:44.922901 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:44.922851 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="32b0bea5-05bb-4c7a-942d-eb057e672324" containerName="payload-pre-processing" Oct 30 01:35:44.926170 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:44.926148 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" Oct 30 01:35:44.959774 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:44.959735 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-666b0b-openshift-default-dockercfg-5rh4p\"" Oct 30 01:35:44.960660 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:44.960636 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg"] Oct 30 01:35:45.004857 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:45.004814 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-istiod-ca-cert\") pod \"e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg\" (UID: \"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab\") " pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" Oct 30 01:35:45.005052 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:45.004867 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-workload-certs\") pod \"e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg\" (UID: \"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab\") " pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" Oct 30 01:35:45.005052 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:45.004971 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-credential-socket\") pod \"e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg\" (UID: \"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab\") " pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" Oct 30 01:35:45.005184 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:45.005153 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-istio-podinfo\") pod \"e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg\" (UID: \"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab\") " pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" Oct 30 01:35:45.005279 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:45.005243 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-istio-data\") pod \"e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg\" (UID: \"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab\") " pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" Oct 30 01:35:45.005345 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:45.005310 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc9xd\" (UniqueName: \"kubernetes.io/projected/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-kube-api-access-tc9xd\") pod \"e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg\" (UID: \"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab\") " pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" Oct 30 01:35:45.005410 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:45.005365 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-istio-envoy\") pod \"e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg\" (UID: \"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab\") " pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" Oct 30 01:35:45.005469 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:45.005435 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-workload-socket\") pod \"e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg\" (UID: \"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab\") " pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" Oct 30 01:35:45.005550 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:45.005519 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-istio-token\") pod \"e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg\" (UID: \"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab\") " pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" Oct 30 01:35:45.114606 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:45.112183 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-istio-data\") pod \"e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg\" (UID: \"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab\") " pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" Oct 30 01:35:45.114606 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:45.112287 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc9xd\" (UniqueName: \"kubernetes.io/projected/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-kube-api-access-tc9xd\") pod \"e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg\" (UID: \"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab\") " pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" Oct 30 01:35:45.114606 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:45.112331 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-istio-envoy\") pod \"e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg\" (UID: \"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab\") " pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" Oct 30 01:35:45.114606 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:45.112400 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-workload-socket\") pod \"e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg\" (UID: \"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab\") " pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" Oct 30 01:35:45.114606 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:45.112502 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-istio-token\") pod \"e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg\" (UID: \"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab\") " pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" Oct 30 01:35:45.114606 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:45.112596 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-istiod-ca-cert\") pod \"e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg\" (UID: \"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab\") " pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" Oct 30 01:35:45.114606 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:45.112624 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-workload-certs\") pod \"e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg\" (UID: \"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab\") " pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" Oct 30 01:35:45.114606 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:45.112665 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-credential-socket\") pod \"e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg\" (UID: \"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab\") " pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" Oct 30 01:35:45.114606 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:45.112844 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-istio-podinfo\") pod \"e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg\" (UID: \"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab\") " pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" Oct 30 01:35:45.122574 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:45.118969 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-workload-socket\") pod \"e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg\" (UID: \"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab\") " pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" Oct 30 01:35:45.122574 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:45.119253 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-istio-data\") pod \"e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg\" (UID: \"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab\") " pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" Oct 30 01:35:45.122574 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:45.120405 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-istiod-ca-cert\") pod \"e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg\" (UID: \"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab\") " pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" Oct 30 01:35:45.122574 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:45.120854 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-workload-certs\") pod \"e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg\" (UID: \"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab\") " pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" Oct 30 01:35:45.122574 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:45.121061 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-credential-socket\") pod \"e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg\" (UID: \"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab\") " pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" Oct 30 01:35:45.243025 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:45.242879 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc9xd\" (UniqueName: \"kubernetes.io/projected/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-kube-api-access-tc9xd\") pod \"e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg\" (UID: \"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab\") " pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" Oct 30 01:35:45.243777 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:45.243715 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-istio-token\") pod \"e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg\" (UID: \"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab\") " pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" Oct 30 01:35:45.244281 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:45.244225 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-istio-envoy\") pod \"e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg\" (UID: \"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab\") " pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" Oct 30 01:35:45.265270 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:45.265193 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-istio-podinfo\") pod \"e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg\" (UID: \"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab\") " pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" Oct 30 01:35:45.547244 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:45.546895 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" Oct 30 01:35:45.911287 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:45.911253 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg"] Oct 30 01:35:45.913008 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:35:45.912975 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfefc3c51_c7f6_4f1c_85dd_2c23ab8a99ab.slice/crio-3ebc97722c49cb06e3dbc45b5a5a2553ea08cf93be2dc8da68f2290d0db5a62a WatchSource:0}: Error finding container 3ebc97722c49cb06e3dbc45b5a5a2553ea08cf93be2dc8da68f2290d0db5a62a: Status 404 returned error can't find the container with id 3ebc97722c49cb06e3dbc45b5a5a2553ea08cf93be2dc8da68f2290d0db5a62a Oct 30 01:35:45.916713 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:45.916664 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:35:45.916814 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:45.916787 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:35:45.916861 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:45.916843 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:35:46.372641 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:46.372392 6221 patch_prober.go:29] interesting pod/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": read tcp 10.134.0.2:34636->10.134.0.86:15021: read: connection reset by peer" start-of-body= Oct 30 01:35:46.372641 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:46.372450 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" podUID="d30042d8-3e81-4825-9afa-9848e0f06141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": read tcp 10.134.0.2:34636->10.134.0.86:15021: read: connection reset by peer" Oct 30 01:35:46.657645 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:46.657602 6221 patch_prober.go:29] interesting pod/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 01:35:46.657839 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:46.657671 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" podUID="60112bce-f58b-467c-8afe-11a1bb92f891" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 01:35:46.754227 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:46.754194 6221 generic.go:356] "Generic (PLEG): container finished" podID="d30042d8-3e81-4825-9afa-9848e0f06141" containerID="3249149ef74bd2d4da3f251e78605238a18942e5d5a4982ee8bc30f368553b2d" exitCode=0 Oct 30 01:35:46.754402 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:46.754270 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" event={"ID":"d30042d8-3e81-4825-9afa-9848e0f06141","Type":"ContainerDied","Data":"3249149ef74bd2d4da3f251e78605238a18942e5d5a4982ee8bc30f368553b2d"} Oct 30 01:35:46.754814 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:46.754791 6221 scope.go:122] "RemoveContainer" containerID="3249149ef74bd2d4da3f251e78605238a18942e5d5a4982ee8bc30f368553b2d" Oct 30 01:35:46.755987 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:46.755953 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" event={"ID":"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab","Type":"ContainerStarted","Data":"3bd4319c2a6fd48570dcd57f73bab2ce1dd6117f44f272b20e2415cdf5f1b2fd"} Oct 30 01:35:46.756127 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:46.756017 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" event={"ID":"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab","Type":"ContainerStarted","Data":"3ebc97722c49cb06e3dbc45b5a5a2553ea08cf93be2dc8da68f2290d0db5a62a"} Oct 30 01:35:46.756474 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:46.756441 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:35:46.756940 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:46.756522 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:35:46.756940 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:46.756597 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:35:46.757636 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:46.757616 6221 generic.go:356] "Generic (PLEG): container finished" podID="60112bce-f58b-467c-8afe-11a1bb92f891" containerID="13cfd9a06c7ca72616f6a22f8316aa683814cc70af429cc4e1542bae0d310df5" exitCode=0 Oct 30 01:35:46.757705 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:46.757663 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" event={"ID":"60112bce-f58b-467c-8afe-11a1bb92f891","Type":"ContainerDied","Data":"13cfd9a06c7ca72616f6a22f8316aa683814cc70af429cc4e1542bae0d310df5"} Oct 30 01:35:46.757966 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:46.757952 6221 scope.go:122] "RemoveContainer" containerID="13cfd9a06c7ca72616f6a22f8316aa683814cc70af429cc4e1542bae0d310df5" Oct 30 01:35:46.759802 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:46.759722 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:35:46.759802 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:46.759796 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:35:46.759890 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:46.759845 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:35:46.853127 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:46.853062 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" podStartSLOduration=2.853049507 podStartE2EDuration="2.853049507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:35:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:35:46.851004572 +0000 UTC m=+1070.429291909" watchObservedRunningTime="2026-10-30 01:35:46.853049507 +0000 UTC m=+1070.431336842" Oct 30 01:35:47.550554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:47.547941 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" Oct 30 01:35:47.560616 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:47.560586 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" Oct 30 01:35:47.772564 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:47.772059 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" event={"ID":"60112bce-f58b-467c-8afe-11a1bb92f891","Type":"ContainerStarted","Data":"b5cb2b99cd2d8726ea07b858d76aa93207c7edf9295615e5d411469b5cf8097a"} Oct 30 01:35:47.775175 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:47.775132 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" event={"ID":"d30042d8-3e81-4825-9afa-9848e0f06141","Type":"ContainerStarted","Data":"c5d2697229be541f56891b6722896fcfe36f617fd174f1fdcffd32a850df5bc8"} Oct 30 01:35:47.775599 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:47.775579 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" Oct 30 01:35:47.776591 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:47.776520 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" Oct 30 01:35:48.227047 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:48.227008 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" Oct 30 01:35:48.227451 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:48.227350 6221 patch_prober.go:29] interesting pod/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 01:35:48.227668 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:48.227646 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" podUID="d30042d8-3e81-4825-9afa-9848e0f06141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 01:35:48.386094 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:48.386064 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" Oct 30 01:35:48.386389 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:48.386369 6221 patch_prober.go:29] interesting pod/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 01:35:48.386502 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:48.386483 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" podUID="60112bce-f58b-467c-8afe-11a1bb92f891" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 01:35:49.227556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:49.226930 6221 patch_prober.go:29] interesting pod/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 01:35:49.227556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:49.227008 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" podUID="d30042d8-3e81-4825-9afa-9848e0f06141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 01:35:49.386595 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:49.386562 6221 patch_prober.go:29] interesting pod/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 01:35:49.386771 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:49.386614 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" podUID="60112bce-f58b-467c-8afe-11a1bb92f891" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 01:35:50.227556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:50.227047 6221 patch_prober.go:29] interesting pod/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 01:35:50.227556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:50.227126 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" podUID="d30042d8-3e81-4825-9afa-9848e0f06141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 01:35:50.385834 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:50.385798 6221 patch_prober.go:29] interesting pod/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 01:35:50.386083 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:50.386052 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" podUID="60112bce-f58b-467c-8afe-11a1bb92f891" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 01:35:51.260068 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:51.257985 6221 patch_prober.go:29] interesting pod/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 01:35:51.260068 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:51.258048 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" podUID="d30042d8-3e81-4825-9afa-9848e0f06141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 01:35:51.391404 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:51.391350 6221 patch_prober.go:29] interesting pod/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 01:35:51.408556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:51.395611 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" podUID="60112bce-f58b-467c-8afe-11a1bb92f891" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 01:35:51.921304 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:35:51.921264 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:35:52.232729 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:52.232631 6221 patch_prober.go:29] interesting pod/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 01:35:52.232982 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:52.232953 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" podUID="d30042d8-3e81-4825-9afa-9848e0f06141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 01:35:52.387228 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:52.387193 6221 patch_prober.go:29] interesting pod/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 01:35:52.387642 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:52.387268 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" podUID="60112bce-f58b-467c-8afe-11a1bb92f891" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 01:35:53.224993 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:53.224951 6221 patch_prober.go:29] interesting pod/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 30 01:35:53.225295 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:53.225263 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" podUID="d30042d8-3e81-4825-9afa-9848e0f06141" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 30 01:35:53.399125 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:53.399089 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" Oct 30 01:35:53.400749 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:53.400721 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" Oct 30 01:35:53.401597 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:53.401573 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" Oct 30 01:35:54.232475 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:54.232442 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" Oct 30 01:35:54.234261 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:54.234238 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" Oct 30 01:35:54.239647 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:54.239626 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" Oct 30 01:35:54.643078 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:54.643045 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a06822-7c4d8ddf5b-xtpl4"] Oct 30 01:35:54.675448 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:54.675416 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a06822-8557d79667-ctpsb"] Oct 30 01:35:58.174162 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:58.174128 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:35:59.643932 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:59.643868 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-a06822-7c4d8ddf5b-xtpl4" podUID="84b9ecd9-6438-40ee-be8b-23558589777f" containerName="payload-processing" containerID="cri-o://fc6ea9c74d6ddb7f4b17ea43d3e4b7083eff33a335baaa529d27ecb921330b56" gracePeriod=25 Oct 30 01:35:59.676320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:59.676271 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a06822-8557d79667-ctpsb" podUID="1200b6f0-d740-4ca0-80b6-3fa083ce7156" containerName="payload-pre-processing" containerID="cri-o://da5860ed0d0929168af90d74c3091720936ff6a378965781880fe3c52c4dc7f6" gracePeriod=25 Oct 30 01:35:59.942663 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:59.942634 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a06822-7c4d8ddf5b-xtpl4" Oct 30 01:35:59.985077 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:59.985055 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a06822-8557d79667-ctpsb" Oct 30 01:35:59.992797 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:59.992764 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84b9ecd9-6438-40ee-be8b-23558589777f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84b9ecd9-6438-40ee-be8b-23558589777f-plugins-config-volume\") pod \"84b9ecd9-6438-40ee-be8b-23558589777f\" (UID: \"84b9ecd9-6438-40ee-be8b-23558589777f\") " Oct 30 01:35:59.992978 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:59.992956 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84b9ecd9-6438-40ee-be8b-23558589777f-kube-api-access-qmwnq\" (UniqueName: \"kubernetes.io/projected/84b9ecd9-6438-40ee-be8b-23558589777f-kube-api-access-qmwnq\") pod \"84b9ecd9-6438-40ee-be8b-23558589777f\" (UID: \"84b9ecd9-6438-40ee-be8b-23558589777f\") " Oct 30 01:35:59.993479 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:59.993453 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84b9ecd9-6438-40ee-be8b-23558589777f-plugins-config-volume" pod "84b9ecd9-6438-40ee-be8b-23558589777f" (UID: "84b9ecd9-6438-40ee-be8b-23558589777f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:35:59.993684 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:59.993665 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84b9ecd9-6438-40ee-be8b-23558589777f-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:35:59.995781 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:35:59.995755 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84b9ecd9-6438-40ee-be8b-23558589777f-kube-api-access-qmwnq" pod "84b9ecd9-6438-40ee-be8b-23558589777f" (UID: "84b9ecd9-6438-40ee-be8b-23558589777f"). InnerVolumeSpecName "kube-api-access-qmwnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:36:00.037213 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:00.037163 6221 generic.go:356] "Generic (PLEG): container finished" podID="1200b6f0-d740-4ca0-80b6-3fa083ce7156" containerID="da5860ed0d0929168af90d74c3091720936ff6a378965781880fe3c52c4dc7f6" exitCode=0 Oct 30 01:36:00.037409 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:00.037228 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a06822-8557d79667-ctpsb" Oct 30 01:36:00.037409 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:00.037247 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a06822-8557d79667-ctpsb" event={"ID":"1200b6f0-d740-4ca0-80b6-3fa083ce7156","Type":"ContainerDied","Data":"da5860ed0d0929168af90d74c3091720936ff6a378965781880fe3c52c4dc7f6"} Oct 30 01:36:00.037409 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:00.037335 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a06822-8557d79667-ctpsb" event={"ID":"1200b6f0-d740-4ca0-80b6-3fa083ce7156","Type":"ContainerDied","Data":"b8543f9726e769eca8e8e0ae58069b502828b8a210cd0278c39191f3e753365b"} Oct 30 01:36:00.037409 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:00.037364 6221 scope.go:122] "RemoveContainer" containerID="da5860ed0d0929168af90d74c3091720936ff6a378965781880fe3c52c4dc7f6" Oct 30 01:36:00.039116 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:00.039084 6221 generic.go:356] "Generic (PLEG): container finished" podID="84b9ecd9-6438-40ee-be8b-23558589777f" containerID="fc6ea9c74d6ddb7f4b17ea43d3e4b7083eff33a335baaa529d27ecb921330b56" exitCode=0 Oct 30 01:36:00.039228 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:00.039127 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a06822-7c4d8ddf5b-xtpl4" Oct 30 01:36:00.039228 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:00.039136 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a06822-7c4d8ddf5b-xtpl4" event={"ID":"84b9ecd9-6438-40ee-be8b-23558589777f","Type":"ContainerDied","Data":"fc6ea9c74d6ddb7f4b17ea43d3e4b7083eff33a335baaa529d27ecb921330b56"} Oct 30 01:36:00.039228 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:00.039176 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a06822-7c4d8ddf5b-xtpl4" event={"ID":"84b9ecd9-6438-40ee-be8b-23558589777f","Type":"ContainerDied","Data":"cb83807bd45e4531a90bec4bf006aa1df06c550b6d9e21916034c722a803c40b"} Oct 30 01:36:00.051380 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:00.051355 6221 scope.go:122] "RemoveContainer" containerID="da5860ed0d0929168af90d74c3091720936ff6a378965781880fe3c52c4dc7f6" Oct 30 01:36:00.051868 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:36:00.051841 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da5860ed0d0929168af90d74c3091720936ff6a378965781880fe3c52c4dc7f6\": container with ID starting with da5860ed0d0929168af90d74c3091720936ff6a378965781880fe3c52c4dc7f6 not found: ID does not exist" containerID="da5860ed0d0929168af90d74c3091720936ff6a378965781880fe3c52c4dc7f6" Oct 30 01:36:00.051974 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:00.051882 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da5860ed0d0929168af90d74c3091720936ff6a378965781880fe3c52c4dc7f6"} err="failed to get container status \"da5860ed0d0929168af90d74c3091720936ff6a378965781880fe3c52c4dc7f6\": rpc error: code = NotFound desc = could not find container \"da5860ed0d0929168af90d74c3091720936ff6a378965781880fe3c52c4dc7f6\": container with ID starting with da5860ed0d0929168af90d74c3091720936ff6a378965781880fe3c52c4dc7f6 not found: ID does not exist" Oct 30 01:36:00.051974 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:00.051908 6221 scope.go:122] "RemoveContainer" containerID="fc6ea9c74d6ddb7f4b17ea43d3e4b7083eff33a335baaa529d27ecb921330b56" Oct 30 01:36:00.061874 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:00.061854 6221 scope.go:122] "RemoveContainer" containerID="fc6ea9c74d6ddb7f4b17ea43d3e4b7083eff33a335baaa529d27ecb921330b56" Oct 30 01:36:00.062212 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:36:00.062187 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc6ea9c74d6ddb7f4b17ea43d3e4b7083eff33a335baaa529d27ecb921330b56\": container with ID starting with fc6ea9c74d6ddb7f4b17ea43d3e4b7083eff33a335baaa529d27ecb921330b56 not found: ID does not exist" containerID="fc6ea9c74d6ddb7f4b17ea43d3e4b7083eff33a335baaa529d27ecb921330b56" Oct 30 01:36:00.062279 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:00.062225 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc6ea9c74d6ddb7f4b17ea43d3e4b7083eff33a335baaa529d27ecb921330b56"} err="failed to get container status \"fc6ea9c74d6ddb7f4b17ea43d3e4b7083eff33a335baaa529d27ecb921330b56\": rpc error: code = NotFound desc = could not find container \"fc6ea9c74d6ddb7f4b17ea43d3e4b7083eff33a335baaa529d27ecb921330b56\": container with ID starting with fc6ea9c74d6ddb7f4b17ea43d3e4b7083eff33a335baaa529d27ecb921330b56 not found: ID does not exist" Oct 30 01:36:00.089205 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:00.089178 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a06822-7c4d8ddf5b-xtpl4"] Oct 30 01:36:00.092253 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:00.092227 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a06822-7c4d8ddf5b-xtpl4"] Oct 30 01:36:00.094703 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:00.094681 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1200b6f0-d740-4ca0-80b6-3fa083ce7156-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1200b6f0-d740-4ca0-80b6-3fa083ce7156-plugins-config-volume\") pod \"1200b6f0-d740-4ca0-80b6-3fa083ce7156\" (UID: \"1200b6f0-d740-4ca0-80b6-3fa083ce7156\") " Oct 30 01:36:00.094934 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:00.094919 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1200b6f0-d740-4ca0-80b6-3fa083ce7156-kube-api-access-rvwd5\" (UniqueName: \"kubernetes.io/projected/1200b6f0-d740-4ca0-80b6-3fa083ce7156-kube-api-access-rvwd5\") pod \"1200b6f0-d740-4ca0-80b6-3fa083ce7156\" (UID: \"1200b6f0-d740-4ca0-80b6-3fa083ce7156\") " Oct 30 01:36:00.095071 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:00.095051 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1200b6f0-d740-4ca0-80b6-3fa083ce7156-plugins-config-volume" pod "1200b6f0-d740-4ca0-80b6-3fa083ce7156" (UID: "1200b6f0-d740-4ca0-80b6-3fa083ce7156"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:36:00.095409 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:00.095388 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmwnq\" (UniqueName: \"kubernetes.io/projected/84b9ecd9-6438-40ee-be8b-23558589777f-kube-api-access-qmwnq\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:36:00.095468 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:00.095414 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1200b6f0-d740-4ca0-80b6-3fa083ce7156-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:36:00.096927 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:00.096908 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1200b6f0-d740-4ca0-80b6-3fa083ce7156-kube-api-access-rvwd5" pod "1200b6f0-d740-4ca0-80b6-3fa083ce7156" (UID: "1200b6f0-d740-4ca0-80b6-3fa083ce7156"). InnerVolumeSpecName "kube-api-access-rvwd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:36:00.196379 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:00.196300 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvwd5\" (UniqueName: \"kubernetes.io/projected/1200b6f0-d740-4ca0-80b6-3fa083ce7156-kube-api-access-rvwd5\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:36:00.383764 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:00.383732 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a06822-8557d79667-ctpsb"] Oct 30 01:36:00.386918 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:00.386893 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a06822-8557d79667-ctpsb"] Oct 30 01:36:00.548176 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:00.548146 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1200b6f0-d740-4ca0-80b6-3fa083ce7156" path="/var/lib/kubelet/pods/1200b6f0-d740-4ca0-80b6-3fa083ce7156/volumes" Oct 30 01:36:00.548679 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:00.548665 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84b9ecd9-6438-40ee-be8b-23558589777f" path="/var/lib/kubelet/pods/84b9ecd9-6438-40ee-be8b-23558589777f/volumes" Oct 30 01:36:01.674930 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:01.674891 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-qd775"] Oct 30 01:36:04.520806 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:04.520775 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-666b0b-75595758bd-ls7kp"] Oct 30 01:36:04.521444 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:04.521428 6221 cpu_manager.go:414] "containerMap: removing container" podUID="84b9ecd9-6438-40ee-be8b-23558589777f" containerName="payload-processing" Oct 30 01:36:04.521504 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:04.521445 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84b9ecd9-6438-40ee-be8b-23558589777f" containerName="payload-processing" Oct 30 01:36:04.521504 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:04.521473 6221 cpu_manager.go:414] "containerMap: removing container" podUID="1200b6f0-d740-4ca0-80b6-3fa083ce7156" containerName="payload-pre-processing" Oct 30 01:36:04.521504 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:04.521479 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1200b6f0-d740-4ca0-80b6-3fa083ce7156" containerName="payload-pre-processing" Oct 30 01:36:04.521615 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:04.521602 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="84b9ecd9-6438-40ee-be8b-23558589777f" containerName="payload-processing" Oct 30 01:36:04.521651 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:04.521616 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="1200b6f0-d740-4ca0-80b6-3fa083ce7156" containerName="payload-pre-processing" Oct 30 01:36:04.525715 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:04.525699 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-666b0b-75595758bd-ls7kp" Oct 30 01:36:04.550221 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:04.550192 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-666b0b-75595758bd-ls7kp"] Oct 30 01:36:04.552503 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:04.552474 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-666b0b-serving-cert\"" Oct 30 01:36:04.592073 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:04.592038 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06a10c0f-09bf-4e42-b580-15b4fc354b59-metrics-certs\") pod \"maas-api-e2e-api-b-666b0b-75595758bd-ls7kp\" (UID: \"06a10c0f-09bf-4e42-b580-15b4fc354b59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-666b0b-75595758bd-ls7kp" Oct 30 01:36:04.592223 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:04.592148 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcwht\" (UniqueName: \"kubernetes.io/projected/06a10c0f-09bf-4e42-b580-15b4fc354b59-kube-api-access-wcwht\") pod \"maas-api-e2e-api-b-666b0b-75595758bd-ls7kp\" (UID: \"06a10c0f-09bf-4e42-b580-15b4fc354b59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-666b0b-75595758bd-ls7kp" Oct 30 01:36:04.592282 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:04.592234 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06a10c0f-09bf-4e42-b580-15b4fc354b59-openshift-service-ca\") pod \"maas-api-e2e-api-b-666b0b-75595758bd-ls7kp\" (UID: \"06a10c0f-09bf-4e42-b580-15b4fc354b59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-666b0b-75595758bd-ls7kp" Oct 30 01:36:04.592335 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:04.592317 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06a10c0f-09bf-4e42-b580-15b4fc354b59-maas-api-tls\") pod \"maas-api-e2e-api-b-666b0b-75595758bd-ls7kp\" (UID: \"06a10c0f-09bf-4e42-b580-15b4fc354b59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-666b0b-75595758bd-ls7kp" Oct 30 01:36:04.694380 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:04.694344 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06a10c0f-09bf-4e42-b580-15b4fc354b59-metrics-certs\") pod \"maas-api-e2e-api-b-666b0b-75595758bd-ls7kp\" (UID: \"06a10c0f-09bf-4e42-b580-15b4fc354b59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-666b0b-75595758bd-ls7kp" Oct 30 01:36:04.694603 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:04.694410 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcwht\" (UniqueName: \"kubernetes.io/projected/06a10c0f-09bf-4e42-b580-15b4fc354b59-kube-api-access-wcwht\") pod \"maas-api-e2e-api-b-666b0b-75595758bd-ls7kp\" (UID: \"06a10c0f-09bf-4e42-b580-15b4fc354b59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-666b0b-75595758bd-ls7kp" Oct 30 01:36:04.694603 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:04.694480 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06a10c0f-09bf-4e42-b580-15b4fc354b59-openshift-service-ca\") pod \"maas-api-e2e-api-b-666b0b-75595758bd-ls7kp\" (UID: \"06a10c0f-09bf-4e42-b580-15b4fc354b59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-666b0b-75595758bd-ls7kp" Oct 30 01:36:04.694603 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:04.694514 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06a10c0f-09bf-4e42-b580-15b4fc354b59-maas-api-tls\") pod \"maas-api-e2e-api-b-666b0b-75595758bd-ls7kp\" (UID: \"06a10c0f-09bf-4e42-b580-15b4fc354b59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-666b0b-75595758bd-ls7kp" Oct 30 01:36:04.695366 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:04.695340 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06a10c0f-09bf-4e42-b580-15b4fc354b59-openshift-service-ca\") pod \"maas-api-e2e-api-b-666b0b-75595758bd-ls7kp\" (UID: \"06a10c0f-09bf-4e42-b580-15b4fc354b59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-666b0b-75595758bd-ls7kp" Oct 30 01:36:04.697173 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:04.697150 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06a10c0f-09bf-4e42-b580-15b4fc354b59-maas-api-tls\") pod \"maas-api-e2e-api-b-666b0b-75595758bd-ls7kp\" (UID: \"06a10c0f-09bf-4e42-b580-15b4fc354b59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-666b0b-75595758bd-ls7kp" Oct 30 01:36:04.698664 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:04.698646 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06a10c0f-09bf-4e42-b580-15b4fc354b59-metrics-certs\") pod \"maas-api-e2e-api-b-666b0b-75595758bd-ls7kp\" (UID: \"06a10c0f-09bf-4e42-b580-15b4fc354b59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-666b0b-75595758bd-ls7kp" Oct 30 01:36:04.714096 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:04.714066 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcwht\" (UniqueName: \"kubernetes.io/projected/06a10c0f-09bf-4e42-b580-15b4fc354b59-kube-api-access-wcwht\") pod \"maas-api-e2e-api-b-666b0b-75595758bd-ls7kp\" (UID: \"06a10c0f-09bf-4e42-b580-15b4fc354b59\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-666b0b-75595758bd-ls7kp" Oct 30 01:36:04.840876 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:04.839960 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-666b0b-75595758bd-ls7kp" Oct 30 01:36:05.185583 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:05.185557 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-666b0b-75595758bd-ls7kp"] Oct 30 01:36:05.187870 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:36:05.187846 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06a10c0f_09bf_4e42_b580_15b4fc354b59.slice/crio-d24264a8761effdae4413b62ffe8d4313a4e7dbecb5a65e548d92524728fa1ad WatchSource:0}: Error finding container d24264a8761effdae4413b62ffe8d4313a4e7dbecb5a65e548d92524728fa1ad: Status 404 returned error can't find the container with id d24264a8761effdae4413b62ffe8d4313a4e7dbecb5a65e548d92524728fa1ad Oct 30 01:36:06.117636 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:06.117601 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-666b0b-75595758bd-ls7kp" event={"ID":"06a10c0f-09bf-4e42-b580-15b4fc354b59","Type":"ContainerStarted","Data":"a3a2888000833f9e9b42b35e974af8b0ceba2c5202ca5b738b37bef2859a757d"} Oct 30 01:36:06.118144 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:06.117661 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-666b0b-75595758bd-ls7kp" event={"ID":"06a10c0f-09bf-4e42-b580-15b4fc354b59","Type":"ContainerStarted","Data":"d24264a8761effdae4413b62ffe8d4313a4e7dbecb5a65e548d92524728fa1ad"} Oct 30 01:36:06.118144 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:06.117760 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-666b0b-75595758bd-ls7kp" Oct 30 01:36:06.158242 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:06.158185 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-666b0b-75595758bd-ls7kp" podStartSLOduration=1.842635851 podStartE2EDuration="2.158172469s" totalImagesPullingTime="315.536618ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:36:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:36:06.157567051 +0000 UTC m=+1089.735854387" watchObservedRunningTime="2026-10-30 01:36:06.158172469 +0000 UTC m=+1089.736459804" Oct 30 01:36:06.691558 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:06.680601 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-qd775" podUID="0ce0d4a4-6037-4aff-b762-f9ed9c0f1816" containerName="payload-pre-processing" containerID="cri-o://32a7d9570e7b6cc0537c7cac808e00eee56b015f3cbd269caacad84198b6eca6" gracePeriod=25 Oct 30 01:36:07.141713 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:07.141672 6221 generic.go:356] "Generic (PLEG): container finished" podID="0ce0d4a4-6037-4aff-b762-f9ed9c0f1816" containerID="32a7d9570e7b6cc0537c7cac808e00eee56b015f3cbd269caacad84198b6eca6" exitCode=0 Oct 30 01:36:07.142170 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:07.141693 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-qd775" event={"ID":"0ce0d4a4-6037-4aff-b762-f9ed9c0f1816","Type":"ContainerDied","Data":"32a7d9570e7b6cc0537c7cac808e00eee56b015f3cbd269caacad84198b6eca6"} Oct 30 01:36:07.263655 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:07.263634 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-qd775" Oct 30 01:36:07.431278 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:07.431244 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ce0d4a4-6037-4aff-b762-f9ed9c0f1816-kube-api-access-9smz8\" (UniqueName: \"kubernetes.io/projected/0ce0d4a4-6037-4aff-b762-f9ed9c0f1816-kube-api-access-9smz8\") pod \"0ce0d4a4-6037-4aff-b762-f9ed9c0f1816\" (UID: \"0ce0d4a4-6037-4aff-b762-f9ed9c0f1816\") " Oct 30 01:36:07.431482 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:07.431397 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ce0d4a4-6037-4aff-b762-f9ed9c0f1816-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ce0d4a4-6037-4aff-b762-f9ed9c0f1816-plugins-config-volume\") pod \"0ce0d4a4-6037-4aff-b762-f9ed9c0f1816\" (UID: \"0ce0d4a4-6037-4aff-b762-f9ed9c0f1816\") " Oct 30 01:36:07.431788 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:07.431757 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ce0d4a4-6037-4aff-b762-f9ed9c0f1816-plugins-config-volume" pod "0ce0d4a4-6037-4aff-b762-f9ed9c0f1816" (UID: "0ce0d4a4-6037-4aff-b762-f9ed9c0f1816"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:36:07.432042 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:07.432026 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ce0d4a4-6037-4aff-b762-f9ed9c0f1816-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:36:07.433344 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:07.433324 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ce0d4a4-6037-4aff-b762-f9ed9c0f1816-kube-api-access-9smz8" pod "0ce0d4a4-6037-4aff-b762-f9ed9c0f1816" (UID: "0ce0d4a4-6037-4aff-b762-f9ed9c0f1816"). InnerVolumeSpecName "kube-api-access-9smz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:36:07.532887 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:07.532862 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9smz8\" (UniqueName: \"kubernetes.io/projected/0ce0d4a4-6037-4aff-b762-f9ed9c0f1816-kube-api-access-9smz8\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:36:08.150202 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:08.150169 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-qd775" event={"ID":"0ce0d4a4-6037-4aff-b762-f9ed9c0f1816","Type":"ContainerDied","Data":"1f02c93aabcba62f0ce2ae772b0b26c347a4fd3e2244f1718c124492580d6bc3"} Oct 30 01:36:08.150710 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:08.150235 6221 scope.go:122] "RemoveContainer" containerID="32a7d9570e7b6cc0537c7cac808e00eee56b015f3cbd269caacad84198b6eca6" Oct 30 01:36:08.150710 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:08.150185 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-qd775" Oct 30 01:36:08.202644 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:08.202613 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-qd775"] Oct 30 01:36:08.214872 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:08.214848 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-278fe7d5-7c7d87757f-qd775"] Oct 30 01:36:08.548377 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:08.548347 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ce0d4a4-6037-4aff-b762-f9ed9c0f1816" path="/var/lib/kubelet/pods/0ce0d4a4-6037-4aff-b762-f9ed9c0f1816/volumes" Oct 30 01:36:12.223560 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:12.223342 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-666b0b-75595758bd-ls7kp" Oct 30 01:36:13.753662 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:13.753629 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8"] Oct 30 01:36:13.754169 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:13.753882 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" podUID="bae879f9-b49c-4f58-a5b3-8c8dd936c89f" containerName="istio-proxy" containerID="cri-o://b81e88138f11f719f0c85a765be028d12c35d1ed664c86c3726c5f5411ac04f5" gracePeriod=30 Oct 30 01:36:16.659015 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:16.658971 6221 patch_prober.go:29] interesting pod/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:36:16.659469 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:16.659031 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" podUID="bae879f9-b49c-4f58-a5b3-8c8dd936c89f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:36:19.266574 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:19.266499 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" Oct 30 01:36:19.286269 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:19.286236 6221 generic.go:356] "Generic (PLEG): container finished" podID="bae879f9-b49c-4f58-a5b3-8c8dd936c89f" containerID="b81e88138f11f719f0c85a765be028d12c35d1ed664c86c3726c5f5411ac04f5" exitCode=0 Oct 30 01:36:19.286463 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:19.286358 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" event={"ID":"bae879f9-b49c-4f58-a5b3-8c8dd936c89f","Type":"ContainerDied","Data":"b81e88138f11f719f0c85a765be028d12c35d1ed664c86c3726c5f5411ac04f5"} Oct 30 01:36:19.286463 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:19.286409 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" event={"ID":"bae879f9-b49c-4f58-a5b3-8c8dd936c89f","Type":"ContainerDied","Data":"120d823e18c013becbbdbc5b8dd910cff277766a87723680afa7fbc4895e1fc6"} Oct 30 01:36:19.286463 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:19.286436 6221 scope.go:122] "RemoveContainer" containerID="b81e88138f11f719f0c85a765be028d12c35d1ed664c86c3726c5f5411ac04f5" Oct 30 01:36:19.286690 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:19.286619 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8" Oct 30 01:36:19.299760 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:19.299730 6221 scope.go:122] "RemoveContainer" containerID="b1210ea546ae404f24da2f0b41fa282e95b95c9575a194d71891569b6c632fc0" Oct 30 01:36:19.313249 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:19.313226 6221 scope.go:122] "RemoveContainer" containerID="b81e88138f11f719f0c85a765be028d12c35d1ed664c86c3726c5f5411ac04f5" Oct 30 01:36:19.314129 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:36:19.314092 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b81e88138f11f719f0c85a765be028d12c35d1ed664c86c3726c5f5411ac04f5\": container with ID starting with b81e88138f11f719f0c85a765be028d12c35d1ed664c86c3726c5f5411ac04f5 not found: ID does not exist" containerID="b81e88138f11f719f0c85a765be028d12c35d1ed664c86c3726c5f5411ac04f5" Oct 30 01:36:19.314247 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:19.314143 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b81e88138f11f719f0c85a765be028d12c35d1ed664c86c3726c5f5411ac04f5"} err="failed to get container status \"b81e88138f11f719f0c85a765be028d12c35d1ed664c86c3726c5f5411ac04f5\": rpc error: code = NotFound desc = could not find container \"b81e88138f11f719f0c85a765be028d12c35d1ed664c86c3726c5f5411ac04f5\": container with ID starting with b81e88138f11f719f0c85a765be028d12c35d1ed664c86c3726c5f5411ac04f5 not found: ID does not exist" Oct 30 01:36:19.314247 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:19.314171 6221 scope.go:122] "RemoveContainer" containerID="b1210ea546ae404f24da2f0b41fa282e95b95c9575a194d71891569b6c632fc0" Oct 30 01:36:19.314569 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:36:19.314516 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1210ea546ae404f24da2f0b41fa282e95b95c9575a194d71891569b6c632fc0\": container with ID starting with b1210ea546ae404f24da2f0b41fa282e95b95c9575a194d71891569b6c632fc0 not found: ID does not exist" containerID="b1210ea546ae404f24da2f0b41fa282e95b95c9575a194d71891569b6c632fc0" Oct 30 01:36:19.314697 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:19.314569 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1210ea546ae404f24da2f0b41fa282e95b95c9575a194d71891569b6c632fc0"} err="failed to get container status \"b1210ea546ae404f24da2f0b41fa282e95b95c9575a194d71891569b6c632fc0\": rpc error: code = NotFound desc = could not find container \"b1210ea546ae404f24da2f0b41fa282e95b95c9575a194d71891569b6c632fc0\": container with ID starting with b1210ea546ae404f24da2f0b41fa282e95b95c9575a194d71891569b6c632fc0 not found: ID does not exist" Oct 30 01:36:19.411605 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:19.386587 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-credential-socket\") pod \"bae879f9-b49c-4f58-a5b3-8c8dd936c89f\" (UID: \"bae879f9-b49c-4f58-a5b3-8c8dd936c89f\") " Oct 30 01:36:19.411605 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:19.386647 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-kube-api-access-sgbcj\" (UniqueName: \"kubernetes.io/projected/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-kube-api-access-sgbcj\") pod \"bae879f9-b49c-4f58-a5b3-8c8dd936c89f\" (UID: \"bae879f9-b49c-4f58-a5b3-8c8dd936c89f\") " Oct 30 01:36:19.411605 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:19.386700 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-istio-token\" (UniqueName: \"kubernetes.io/projected/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-istio-token\") pod \"bae879f9-b49c-4f58-a5b3-8c8dd936c89f\" (UID: \"bae879f9-b49c-4f58-a5b3-8c8dd936c89f\") " Oct 30 01:36:19.411605 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:19.386768 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-istiod-ca-cert\") pod \"bae879f9-b49c-4f58-a5b3-8c8dd936c89f\" (UID: \"bae879f9-b49c-4f58-a5b3-8c8dd936c89f\") " Oct 30 01:36:19.411605 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:19.386801 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-istio-podinfo\") pod \"bae879f9-b49c-4f58-a5b3-8c8dd936c89f\" (UID: \"bae879f9-b49c-4f58-a5b3-8c8dd936c89f\") " Oct 30 01:36:19.411605 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:19.386883 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-istio-envoy\") pod \"bae879f9-b49c-4f58-a5b3-8c8dd936c89f\" (UID: \"bae879f9-b49c-4f58-a5b3-8c8dd936c89f\") " Oct 30 01:36:19.411605 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:19.386910 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-istio-data\") pod \"bae879f9-b49c-4f58-a5b3-8c8dd936c89f\" (UID: \"bae879f9-b49c-4f58-a5b3-8c8dd936c89f\") " Oct 30 01:36:19.411605 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:19.386975 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-workload-socket\") pod \"bae879f9-b49c-4f58-a5b3-8c8dd936c89f\" (UID: \"bae879f9-b49c-4f58-a5b3-8c8dd936c89f\") " Oct 30 01:36:19.411605 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:19.403857 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-workload-certs\") pod \"bae879f9-b49c-4f58-a5b3-8c8dd936c89f\" (UID: \"bae879f9-b49c-4f58-a5b3-8c8dd936c89f\") " Oct 30 01:36:19.411605 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:19.404805 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-workload-socket" pod "bae879f9-b49c-4f58-a5b3-8c8dd936c89f" (UID: "bae879f9-b49c-4f58-a5b3-8c8dd936c89f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:36:19.411605 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:19.405439 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:36:19.411605 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:19.405699 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-workload-certs" pod "bae879f9-b49c-4f58-a5b3-8c8dd936c89f" (UID: "bae879f9-b49c-4f58-a5b3-8c8dd936c89f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:36:19.411605 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:19.406082 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-istiod-ca-cert" pod "bae879f9-b49c-4f58-a5b3-8c8dd936c89f" (UID: "bae879f9-b49c-4f58-a5b3-8c8dd936c89f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:36:19.413688 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:19.411628 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-istio-data" pod "bae879f9-b49c-4f58-a5b3-8c8dd936c89f" (UID: "bae879f9-b49c-4f58-a5b3-8c8dd936c89f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:36:19.413688 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:19.412294 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-credential-socket" pod "bae879f9-b49c-4f58-a5b3-8c8dd936c89f" (UID: "bae879f9-b49c-4f58-a5b3-8c8dd936c89f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:36:19.485687 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:19.483952 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-istio-envoy" pod "bae879f9-b49c-4f58-a5b3-8c8dd936c89f" (UID: "bae879f9-b49c-4f58-a5b3-8c8dd936c89f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:36:19.485687 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:19.484000 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-istio-token" pod "bae879f9-b49c-4f58-a5b3-8c8dd936c89f" (UID: "bae879f9-b49c-4f58-a5b3-8c8dd936c89f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:36:19.485687 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:19.484041 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-kube-api-access-sgbcj" pod "bae879f9-b49c-4f58-a5b3-8c8dd936c89f" (UID: "bae879f9-b49c-4f58-a5b3-8c8dd936c89f"). InnerVolumeSpecName "kube-api-access-sgbcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:36:19.489959 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:19.489927 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-istio-podinfo" pod "bae879f9-b49c-4f58-a5b3-8c8dd936c89f" (UID: "bae879f9-b49c-4f58-a5b3-8c8dd936c89f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:36:19.507042 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:19.507014 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:36:19.507042 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:19.507041 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:36:19.507188 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:19.507051 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgbcj\" (UniqueName: \"kubernetes.io/projected/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-kube-api-access-sgbcj\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:36:19.507188 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:19.507061 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:36:19.507188 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:19.507070 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:36:19.507188 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:19.507077 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:36:19.507188 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:19.507085 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:36:19.507188 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:19.507092 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bae879f9-b49c-4f58-a5b3-8c8dd936c89f-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:36:19.633501 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:19.633470 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8"] Oct 30 01:36:19.645783 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:19.645755 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-4d38e5-openshift-default-84b88f497c-gv6x8"] Oct 30 01:36:20.548484 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:20.548449 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bae879f9-b49c-4f58-a5b3-8c8dd936c89f" path="/var/lib/kubelet/pods/bae879f9-b49c-4f58-a5b3-8c8dd936c89f/volumes" Oct 30 01:36:21.744704 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:36:21.728159 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:36:24.733925 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:24.733888 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-278fe7d5/e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx"] Oct 30 01:36:31.545515 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:31.545482 6221 generic.go:356] "Generic (PLEG): container finished" podID="fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab" containerID="3bd4319c2a6fd48570dcd57f73bab2ce1dd6117f44f272b20e2415cdf5f1b2fd" exitCode=0 Oct 30 01:36:31.545998 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:31.545558 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" event={"ID":"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab","Type":"ContainerDied","Data":"3bd4319c2a6fd48570dcd57f73bab2ce1dd6117f44f272b20e2415cdf5f1b2fd"} Oct 30 01:36:31.545998 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:31.545986 6221 scope.go:122] "RemoveContainer" containerID="3bd4319c2a6fd48570dcd57f73bab2ce1dd6117f44f272b20e2415cdf5f1b2fd" Oct 30 01:36:31.547413 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:31.547387 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:36:31.547497 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:31.547452 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:36:31.547497 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:31.547485 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:36:32.553088 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:32.553055 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" event={"ID":"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab","Type":"ContainerStarted","Data":"28b4e7545449943383c9f91713bca136811f3b234c81bd81a7c4c27d4b882c4a"} Oct 30 01:36:32.776151 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:32.776116 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" Oct 30 01:36:33.547401 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:33.547370 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" Oct 30 01:36:33.548212 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:33.548189 6221 patch_prober.go:29] interesting pod/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 01:36:33.548326 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:33.548242 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" podUID="fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 01:36:34.554906 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:34.554871 6221 patch_prober.go:29] interesting pod/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 01:36:34.555309 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:34.554943 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" podUID="fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 01:36:34.763169 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:34.763141 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f09934-98df65955-8cwfc"] Oct 30 01:36:34.763759 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:34.763745 6221 cpu_manager.go:414] "containerMap: removing container" podUID="bae879f9-b49c-4f58-a5b3-8c8dd936c89f" containerName="istio-proxy" Oct 30 01:36:34.763813 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:34.763760 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bae879f9-b49c-4f58-a5b3-8c8dd936c89f" containerName="istio-proxy" Oct 30 01:36:34.763813 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:34.763774 6221 cpu_manager.go:414] "containerMap: removing container" podUID="0ce0d4a4-6037-4aff-b762-f9ed9c0f1816" containerName="payload-pre-processing" Oct 30 01:36:34.763813 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:34.763780 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ce0d4a4-6037-4aff-b762-f9ed9c0f1816" containerName="payload-pre-processing" Oct 30 01:36:34.763813 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:34.763796 6221 cpu_manager.go:414] "containerMap: removing container" podUID="bae879f9-b49c-4f58-a5b3-8c8dd936c89f" containerName="istio-proxy" Oct 30 01:36:34.763813 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:34.763801 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bae879f9-b49c-4f58-a5b3-8c8dd936c89f" containerName="istio-proxy" Oct 30 01:36:34.763966 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:34.763915 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="bae879f9-b49c-4f58-a5b3-8c8dd936c89f" containerName="istio-proxy" Oct 30 01:36:34.763966 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:34.763925 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ce0d4a4-6037-4aff-b762-f9ed9c0f1816" containerName="payload-pre-processing" Oct 30 01:36:34.763966 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:34.763934 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="bae879f9-b49c-4f58-a5b3-8c8dd936c89f" containerName="istio-proxy" Oct 30 01:36:34.774810 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:34.774770 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f09934-98df65955-8cwfc" Oct 30 01:36:34.787129 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:34.786319 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f09934-98df65955-8cwfc"] Oct 30 01:36:34.880610 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:34.853058 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-f09934-dockercfg-kfxb7\"" Oct 30 01:36:34.880610 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:34.856696 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-f09934\"" Oct 30 01:36:34.924555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:34.889576 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs8r7\" (UniqueName: \"kubernetes.io/projected/8ece0817-69ac-4fa5-a10b-67b88b368b51-kube-api-access-xs8r7\") pod \"payload-pre-processing-e2e-api-a-f09934-98df65955-8cwfc\" (UID: \"8ece0817-69ac-4fa5-a10b-67b88b368b51\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f09934-98df65955-8cwfc" Oct 30 01:36:34.924555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:34.889797 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ece0817-69ac-4fa5-a10b-67b88b368b51-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f09934-98df65955-8cwfc\" (UID: \"8ece0817-69ac-4fa5-a10b-67b88b368b51\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f09934-98df65955-8cwfc" Oct 30 01:36:35.044596 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:35.018724 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs8r7\" (UniqueName: \"kubernetes.io/projected/8ece0817-69ac-4fa5-a10b-67b88b368b51-kube-api-access-xs8r7\") pod \"payload-pre-processing-e2e-api-a-f09934-98df65955-8cwfc\" (UID: \"8ece0817-69ac-4fa5-a10b-67b88b368b51\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f09934-98df65955-8cwfc" Oct 30 01:36:35.044596 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:35.018993 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ece0817-69ac-4fa5-a10b-67b88b368b51-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f09934-98df65955-8cwfc\" (UID: \"8ece0817-69ac-4fa5-a10b-67b88b368b51\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f09934-98df65955-8cwfc" Oct 30 01:36:35.044596 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:35.020007 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ece0817-69ac-4fa5-a10b-67b88b368b51-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f09934-98df65955-8cwfc\" (UID: \"8ece0817-69ac-4fa5-a10b-67b88b368b51\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f09934-98df65955-8cwfc" Oct 30 01:36:35.114568 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:35.085558 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs8r7\" (UniqueName: \"kubernetes.io/projected/8ece0817-69ac-4fa5-a10b-67b88b368b51-kube-api-access-xs8r7\") pod \"payload-pre-processing-e2e-api-a-f09934-98df65955-8cwfc\" (UID: \"8ece0817-69ac-4fa5-a10b-67b88b368b51\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f09934-98df65955-8cwfc" Oct 30 01:36:35.114568 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:35.097814 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f09934-98df65955-8cwfc" Oct 30 01:36:35.148741 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:35.148649 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:36:35.549448 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:35.548980 6221 patch_prober.go:29] interesting pod/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 01:36:35.549448 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:35.549041 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" podUID="fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 01:36:35.836831 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:35.836802 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f09934-98df65955-8cwfc"] Oct 30 01:36:35.839912 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:36:35.839879 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ece0817_69ac_4fa5_a10b_67b88b368b51.slice/crio-3ee6b88a33d9fc3f7391b7aeec41d51f625e27f0f27e2459f1a1d88c7b91ae2f WatchSource:0}: Error finding container 3ee6b88a33d9fc3f7391b7aeec41d51f625e27f0f27e2459f1a1d88c7b91ae2f: Status 404 returned error can't find the container with id 3ee6b88a33d9fc3f7391b7aeec41d51f625e27f0f27e2459f1a1d88c7b91ae2f Oct 30 01:36:36.307615 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:36.304622 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-288c8f-5db9bddc67-hz2dc"] Oct 30 01:36:36.572741 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:36.566816 6221 patch_prober.go:29] interesting pod/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 01:36:36.572741 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:36.566874 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" podUID="fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 01:36:36.609366 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:36.603977 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f09934-98df65955-8cwfc" event={"ID":"8ece0817-69ac-4fa5-a10b-67b88b368b51","Type":"ContainerStarted","Data":"be4a2271e9173ee5d0c069d7ca596b3adde0b915fc1c8eedb7057d2582ab2a39"} Oct 30 01:36:36.609366 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:36.604039 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f09934-98df65955-8cwfc" event={"ID":"8ece0817-69ac-4fa5-a10b-67b88b368b51","Type":"ContainerStarted","Data":"3ee6b88a33d9fc3f7391b7aeec41d51f625e27f0f27e2459f1a1d88c7b91ae2f"} Oct 30 01:36:36.609366 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:36.604990 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f09934-98df65955-8cwfc" Oct 30 01:36:36.648506 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:36.648445 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f09934-98df65955-8cwfc" podStartSLOduration=2.648426368 podStartE2EDuration="2.648426368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:36:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:36:36.646030966 +0000 UTC m=+1120.224318303" watchObservedRunningTime="2026-10-30 01:36:36.648426368 +0000 UTC m=+1120.226713704" Oct 30 01:36:37.548250 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:37.548212 6221 patch_prober.go:29] interesting pod/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 01:36:37.548772 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:37.548288 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" podUID="fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 01:36:38.644188 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:38.643386 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" Oct 30 01:36:38.655810 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:38.655778 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" Oct 30 01:36:39.595177 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:39.595142 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-97c137-7c4fbd7b9b-rh7wf"] Oct 30 01:36:39.627712 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:39.627677 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-97c137-64d87865f-d68pz"] Oct 30 01:36:39.756475 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:39.756433 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-278fe7d5/e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx" podUID="03e0644c-f8df-4da7-9414-ff2bb32e2855" containerName="main" containerID="cri-o://3316e44d699d26a5cbb669f29bb323ea71235d206dfbe3d93770b3502d76cc4d" gracePeriod=45 Oct 30 01:36:40.105832 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:40.105807 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-278fe7d5/e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx" Oct 30 01:36:40.231550 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:40.231457 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03e0644c-f8df-4da7-9414-ff2bb32e2855-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03e0644c-f8df-4da7-9414-ff2bb32e2855-model-cache\") pod \"03e0644c-f8df-4da7-9414-ff2bb32e2855\" (UID: \"03e0644c-f8df-4da7-9414-ff2bb32e2855\") " Oct 30 01:36:40.231550 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:40.231505 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03e0644c-f8df-4da7-9414-ff2bb32e2855-tls-certs\" (UniqueName: \"kubernetes.io/secret/03e0644c-f8df-4da7-9414-ff2bb32e2855-tls-certs\") pod \"03e0644c-f8df-4da7-9414-ff2bb32e2855\" (UID: \"03e0644c-f8df-4da7-9414-ff2bb32e2855\") " Oct 30 01:36:40.231773 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:40.231630 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03e0644c-f8df-4da7-9414-ff2bb32e2855-dshm\" (UniqueName: \"kubernetes.io/empty-dir/03e0644c-f8df-4da7-9414-ff2bb32e2855-dshm\") pod \"03e0644c-f8df-4da7-9414-ff2bb32e2855\" (UID: \"03e0644c-f8df-4da7-9414-ff2bb32e2855\") " Oct 30 01:36:40.231773 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:40.231656 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03e0644c-f8df-4da7-9414-ff2bb32e2855-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03e0644c-f8df-4da7-9414-ff2bb32e2855-tmp-dir\") pod \"03e0644c-f8df-4da7-9414-ff2bb32e2855\" (UID: \"03e0644c-f8df-4da7-9414-ff2bb32e2855\") " Oct 30 01:36:40.231773 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:40.231716 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03e0644c-f8df-4da7-9414-ff2bb32e2855-home\" (UniqueName: \"kubernetes.io/empty-dir/03e0644c-f8df-4da7-9414-ff2bb32e2855-home\") pod \"03e0644c-f8df-4da7-9414-ff2bb32e2855\" (UID: \"03e0644c-f8df-4da7-9414-ff2bb32e2855\") " Oct 30 01:36:40.231773 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:40.231714 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e0644c-f8df-4da7-9414-ff2bb32e2855-model-cache" pod "03e0644c-f8df-4da7-9414-ff2bb32e2855" (UID: "03e0644c-f8df-4da7-9414-ff2bb32e2855"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:36:40.231929 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:40.231827 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03e0644c-f8df-4da7-9414-ff2bb32e2855-kube-api-access-rxcml\" (UniqueName: \"kubernetes.io/projected/03e0644c-f8df-4da7-9414-ff2bb32e2855-kube-api-access-rxcml\") pod \"03e0644c-f8df-4da7-9414-ff2bb32e2855\" (UID: \"03e0644c-f8df-4da7-9414-ff2bb32e2855\") " Oct 30 01:36:40.231929 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:40.231911 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e0644c-f8df-4da7-9414-ff2bb32e2855-tmp-dir" pod "03e0644c-f8df-4da7-9414-ff2bb32e2855" (UID: "03e0644c-f8df-4da7-9414-ff2bb32e2855"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:36:40.232036 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:40.232012 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e0644c-f8df-4da7-9414-ff2bb32e2855-home" pod "03e0644c-f8df-4da7-9414-ff2bb32e2855" (UID: "03e0644c-f8df-4da7-9414-ff2bb32e2855"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:36:40.232427 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:40.232405 6221 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03e0644c-f8df-4da7-9414-ff2bb32e2855-home\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:36:40.232427 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:40.232427 6221 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03e0644c-f8df-4da7-9414-ff2bb32e2855-model-cache\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:36:40.232618 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:40.232436 6221 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03e0644c-f8df-4da7-9414-ff2bb32e2855-tmp-dir\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:36:40.233764 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:40.233738 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03e0644c-f8df-4da7-9414-ff2bb32e2855-tls-certs" pod "03e0644c-f8df-4da7-9414-ff2bb32e2855" (UID: "03e0644c-f8df-4da7-9414-ff2bb32e2855"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:36:40.233836 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:40.233747 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03e0644c-f8df-4da7-9414-ff2bb32e2855-dshm" pod "03e0644c-f8df-4da7-9414-ff2bb32e2855" (UID: "03e0644c-f8df-4da7-9414-ff2bb32e2855"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:36:40.233876 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:40.233856 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03e0644c-f8df-4da7-9414-ff2bb32e2855-kube-api-access-rxcml" pod "03e0644c-f8df-4da7-9414-ff2bb32e2855" (UID: "03e0644c-f8df-4da7-9414-ff2bb32e2855"). InnerVolumeSpecName "kube-api-access-rxcml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:36:40.349220 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:40.349182 6221 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03e0644c-f8df-4da7-9414-ff2bb32e2855-dshm\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:36:40.349455 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:40.349437 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxcml\" (UniqueName: \"kubernetes.io/projected/03e0644c-f8df-4da7-9414-ff2bb32e2855-kube-api-access-rxcml\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:36:40.349566 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:40.349551 6221 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03e0644c-f8df-4da7-9414-ff2bb32e2855-tls-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:36:40.636899 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:40.636857 6221 generic.go:356] "Generic (PLEG): container finished" podID="03e0644c-f8df-4da7-9414-ff2bb32e2855" containerID="3316e44d699d26a5cbb669f29bb323ea71235d206dfbe3d93770b3502d76cc4d" exitCode=0 Oct 30 01:36:40.637082 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:40.636903 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-278fe7d5/e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx" Oct 30 01:36:40.637082 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:40.636906 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-278fe7d5/e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx" event={"ID":"03e0644c-f8df-4da7-9414-ff2bb32e2855","Type":"ContainerDied","Data":"3316e44d699d26a5cbb669f29bb323ea71235d206dfbe3d93770b3502d76cc4d"} Oct 30 01:36:40.637082 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:40.636952 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-278fe7d5/e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx" event={"ID":"03e0644c-f8df-4da7-9414-ff2bb32e2855","Type":"ContainerDied","Data":"f2abce721efccda7b5a28dcbe14ade3408558a8b9d74352c294c35d550e9af5d"} Oct 30 01:36:40.637082 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:40.636975 6221 scope.go:122] "RemoveContainer" containerID="3316e44d699d26a5cbb669f29bb323ea71235d206dfbe3d93770b3502d76cc4d" Oct 30 01:36:40.647254 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:40.647235 6221 scope.go:122] "RemoveContainer" containerID="3316e44d699d26a5cbb669f29bb323ea71235d206dfbe3d93770b3502d76cc4d" Oct 30 01:36:40.647561 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:36:40.647520 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3316e44d699d26a5cbb669f29bb323ea71235d206dfbe3d93770b3502d76cc4d\": container with ID starting with 3316e44d699d26a5cbb669f29bb323ea71235d206dfbe3d93770b3502d76cc4d not found: ID does not exist" containerID="3316e44d699d26a5cbb669f29bb323ea71235d206dfbe3d93770b3502d76cc4d" Oct 30 01:36:40.647649 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:40.647572 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3316e44d699d26a5cbb669f29bb323ea71235d206dfbe3d93770b3502d76cc4d"} err="failed to get container status \"3316e44d699d26a5cbb669f29bb323ea71235d206dfbe3d93770b3502d76cc4d\": rpc error: code = NotFound desc = could not find container \"3316e44d699d26a5cbb669f29bb323ea71235d206dfbe3d93770b3502d76cc4d\": container with ID starting with 3316e44d699d26a5cbb669f29bb323ea71235d206dfbe3d93770b3502d76cc4d not found: ID does not exist" Oct 30 01:36:40.688227 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:40.688195 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-278fe7d5/e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx"] Oct 30 01:36:40.700269 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:40.700242 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-278fe7d5/e2e-lifecycle-model-278fe7d5-kserve-df84bccc7-smmbx"] Oct 30 01:36:41.305431 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:41.305383 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-288c8f-5db9bddc67-hz2dc" podUID="0c793208-af28-4c73-a6de-9ea3f69e80d6" containerName="payload-pre-processing" containerID="cri-o://0728577adfc60d8fe122811fd71ceb0736812f1928ccf48c99877f4f728b0e3e" gracePeriod=25 Oct 30 01:36:41.586261 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:41.586232 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-288c8f-5db9bddc67-hz2dc" Oct 30 01:36:41.645438 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:41.645403 6221 generic.go:356] "Generic (PLEG): container finished" podID="0c793208-af28-4c73-a6de-9ea3f69e80d6" containerID="0728577adfc60d8fe122811fd71ceb0736812f1928ccf48c99877f4f728b0e3e" exitCode=0 Oct 30 01:36:41.645633 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:41.645463 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-288c8f-5db9bddc67-hz2dc" Oct 30 01:36:41.645633 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:41.645494 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-288c8f-5db9bddc67-hz2dc" event={"ID":"0c793208-af28-4c73-a6de-9ea3f69e80d6","Type":"ContainerDied","Data":"0728577adfc60d8fe122811fd71ceb0736812f1928ccf48c99877f4f728b0e3e"} Oct 30 01:36:41.645633 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:41.645583 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-288c8f-5db9bddc67-hz2dc" event={"ID":"0c793208-af28-4c73-a6de-9ea3f69e80d6","Type":"ContainerDied","Data":"87fb890724504bd9045e7629f18009d8f42e5dcd9be1a3617c3cc8345458027d"} Oct 30 01:36:41.645633 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:41.645618 6221 scope.go:122] "RemoveContainer" containerID="0728577adfc60d8fe122811fd71ceb0736812f1928ccf48c99877f4f728b0e3e" Oct 30 01:36:41.656684 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:41.656663 6221 scope.go:122] "RemoveContainer" containerID="0728577adfc60d8fe122811fd71ceb0736812f1928ccf48c99877f4f728b0e3e" Oct 30 01:36:41.656968 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:36:41.656939 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0728577adfc60d8fe122811fd71ceb0736812f1928ccf48c99877f4f728b0e3e\": container with ID starting with 0728577adfc60d8fe122811fd71ceb0736812f1928ccf48c99877f4f728b0e3e not found: ID does not exist" containerID="0728577adfc60d8fe122811fd71ceb0736812f1928ccf48c99877f4f728b0e3e" Oct 30 01:36:41.657068 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:41.656976 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0728577adfc60d8fe122811fd71ceb0736812f1928ccf48c99877f4f728b0e3e"} err="failed to get container status \"0728577adfc60d8fe122811fd71ceb0736812f1928ccf48c99877f4f728b0e3e\": rpc error: code = NotFound desc = could not find container \"0728577adfc60d8fe122811fd71ceb0736812f1928ccf48c99877f4f728b0e3e\": container with ID starting with 0728577adfc60d8fe122811fd71ceb0736812f1928ccf48c99877f4f728b0e3e not found: ID does not exist" Oct 30 01:36:41.667714 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:41.667693 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c793208-af28-4c73-a6de-9ea3f69e80d6-kube-api-access-nxnnv\" (UniqueName: \"kubernetes.io/projected/0c793208-af28-4c73-a6de-9ea3f69e80d6-kube-api-access-nxnnv\") pod \"0c793208-af28-4c73-a6de-9ea3f69e80d6\" (UID: \"0c793208-af28-4c73-a6de-9ea3f69e80d6\") " Oct 30 01:36:41.667910 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:41.667892 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c793208-af28-4c73-a6de-9ea3f69e80d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c793208-af28-4c73-a6de-9ea3f69e80d6-plugins-config-volume\") pod \"0c793208-af28-4c73-a6de-9ea3f69e80d6\" (UID: \"0c793208-af28-4c73-a6de-9ea3f69e80d6\") " Oct 30 01:36:41.668302 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:41.668276 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c793208-af28-4c73-a6de-9ea3f69e80d6-plugins-config-volume" pod "0c793208-af28-4c73-a6de-9ea3f69e80d6" (UID: "0c793208-af28-4c73-a6de-9ea3f69e80d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:36:41.668613 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:41.668519 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c793208-af28-4c73-a6de-9ea3f69e80d6-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:36:41.669803 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:41.669782 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c793208-af28-4c73-a6de-9ea3f69e80d6-kube-api-access-nxnnv" pod "0c793208-af28-4c73-a6de-9ea3f69e80d6" (UID: "0c793208-af28-4c73-a6de-9ea3f69e80d6"). InnerVolumeSpecName "kube-api-access-nxnnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:36:41.770334 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:41.770303 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxnnv\" (UniqueName: \"kubernetes.io/projected/0c793208-af28-4c73-a6de-9ea3f69e80d6-kube-api-access-nxnnv\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:36:42.031511 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:42.031481 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-288c8f-5db9bddc67-hz2dc"] Oct 30 01:36:42.061339 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:42.061294 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-288c8f-5db9bddc67-hz2dc"] Oct 30 01:36:42.548169 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:42.548133 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03e0644c-f8df-4da7-9414-ff2bb32e2855" path="/var/lib/kubelet/pods/03e0644c-f8df-4da7-9414-ff2bb32e2855/volumes" Oct 30 01:36:42.548598 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:42.548583 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c793208-af28-4c73-a6de-9ea3f69e80d6" path="/var/lib/kubelet/pods/0c793208-af28-4c73-a6de-9ea3f69e80d6/volumes" Oct 30 01:36:44.595692 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:44.595649 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-97c137-7c4fbd7b9b-rh7wf" podUID="6d161227-4d8e-49b1-b4e3-308d91f85588" containerName="payload-processing" containerID="cri-o://3faebd6e277c3480b4230051f4cbf49d8a48b1cb42fb4975b51ecf7af0f6fc9d" gracePeriod=25 Oct 30 01:36:44.628325 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:44.628290 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-97c137-64d87865f-d68pz" podUID="d280af82-ee8f-4f27-8cbb-2e7fbd80b5d1" containerName="payload-pre-processing" containerID="cri-o://4966adbefe210ada5c23be855118b62f1ed74d079638bb32ff3079dad5b01c75" gracePeriod=25 Oct 30 01:36:44.726820 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:44.726788 6221 generic.go:356] "Generic (PLEG): container finished" podID="6d161227-4d8e-49b1-b4e3-308d91f85588" containerID="3faebd6e277c3480b4230051f4cbf49d8a48b1cb42fb4975b51ecf7af0f6fc9d" exitCode=0 Oct 30 01:36:44.726971 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:44.726858 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-97c137-7c4fbd7b9b-rh7wf" event={"ID":"6d161227-4d8e-49b1-b4e3-308d91f85588","Type":"ContainerDied","Data":"3faebd6e277c3480b4230051f4cbf49d8a48b1cb42fb4975b51ecf7af0f6fc9d"} Oct 30 01:36:44.919178 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:44.919150 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-97c137-7c4fbd7b9b-rh7wf" Oct 30 01:36:44.971845 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:44.971817 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-97c137-64d87865f-d68pz" Oct 30 01:36:45.027891 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:45.027860 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d161227-4d8e-49b1-b4e3-308d91f85588-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d161227-4d8e-49b1-b4e3-308d91f85588-plugins-config-volume\") pod \"6d161227-4d8e-49b1-b4e3-308d91f85588\" (UID: \"6d161227-4d8e-49b1-b4e3-308d91f85588\") " Oct 30 01:36:45.028159 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:45.028136 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d161227-4d8e-49b1-b4e3-308d91f85588-kube-api-access-p8hd4\" (UniqueName: \"kubernetes.io/projected/6d161227-4d8e-49b1-b4e3-308d91f85588-kube-api-access-p8hd4\") pod \"6d161227-4d8e-49b1-b4e3-308d91f85588\" (UID: \"6d161227-4d8e-49b1-b4e3-308d91f85588\") " Oct 30 01:36:45.028280 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:45.028261 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d161227-4d8e-49b1-b4e3-308d91f85588-plugins-config-volume" pod "6d161227-4d8e-49b1-b4e3-308d91f85588" (UID: "6d161227-4d8e-49b1-b4e3-308d91f85588"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:36:45.028683 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:45.028661 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d161227-4d8e-49b1-b4e3-308d91f85588-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:36:45.030424 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:45.030393 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d161227-4d8e-49b1-b4e3-308d91f85588-kube-api-access-p8hd4" pod "6d161227-4d8e-49b1-b4e3-308d91f85588" (UID: "6d161227-4d8e-49b1-b4e3-308d91f85588"). InnerVolumeSpecName "kube-api-access-p8hd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:36:45.129840 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:45.129805 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d280af82-ee8f-4f27-8cbb-2e7fbd80b5d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d280af82-ee8f-4f27-8cbb-2e7fbd80b5d1-plugins-config-volume\") pod \"d280af82-ee8f-4f27-8cbb-2e7fbd80b5d1\" (UID: \"d280af82-ee8f-4f27-8cbb-2e7fbd80b5d1\") " Oct 30 01:36:45.130066 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:45.130054 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d280af82-ee8f-4f27-8cbb-2e7fbd80b5d1-kube-api-access-48rz8\" (UniqueName: \"kubernetes.io/projected/d280af82-ee8f-4f27-8cbb-2e7fbd80b5d1-kube-api-access-48rz8\") pod \"d280af82-ee8f-4f27-8cbb-2e7fbd80b5d1\" (UID: \"d280af82-ee8f-4f27-8cbb-2e7fbd80b5d1\") " Oct 30 01:36:45.130181 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:45.130157 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d280af82-ee8f-4f27-8cbb-2e7fbd80b5d1-plugins-config-volume" pod "d280af82-ee8f-4f27-8cbb-2e7fbd80b5d1" (UID: "d280af82-ee8f-4f27-8cbb-2e7fbd80b5d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:36:45.130480 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:45.130465 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d280af82-ee8f-4f27-8cbb-2e7fbd80b5d1-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:36:45.130575 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:45.130482 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8hd4\" (UniqueName: \"kubernetes.io/projected/6d161227-4d8e-49b1-b4e3-308d91f85588-kube-api-access-p8hd4\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:36:45.132070 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:45.132047 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d280af82-ee8f-4f27-8cbb-2e7fbd80b5d1-kube-api-access-48rz8" pod "d280af82-ee8f-4f27-8cbb-2e7fbd80b5d1" (UID: "d280af82-ee8f-4f27-8cbb-2e7fbd80b5d1"). InnerVolumeSpecName "kube-api-access-48rz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:36:45.231886 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:45.231810 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48rz8\" (UniqueName: \"kubernetes.io/projected/d280af82-ee8f-4f27-8cbb-2e7fbd80b5d1-kube-api-access-48rz8\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:36:45.735983 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:45.735944 6221 generic.go:356] "Generic (PLEG): container finished" podID="d280af82-ee8f-4f27-8cbb-2e7fbd80b5d1" containerID="4966adbefe210ada5c23be855118b62f1ed74d079638bb32ff3079dad5b01c75" exitCode=0 Oct 30 01:36:45.736479 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:45.735991 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-97c137-64d87865f-d68pz" Oct 30 01:36:45.736479 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:45.736037 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-97c137-64d87865f-d68pz" event={"ID":"d280af82-ee8f-4f27-8cbb-2e7fbd80b5d1","Type":"ContainerDied","Data":"4966adbefe210ada5c23be855118b62f1ed74d079638bb32ff3079dad5b01c75"} Oct 30 01:36:45.736479 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:45.736089 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-97c137-64d87865f-d68pz" event={"ID":"d280af82-ee8f-4f27-8cbb-2e7fbd80b5d1","Type":"ContainerDied","Data":"399d48f43fba5ef1ea2888f8df119d9226abdafa1e322fe626586be34f881eb4"} Oct 30 01:36:45.736479 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:45.736114 6221 scope.go:122] "RemoveContainer" containerID="4966adbefe210ada5c23be855118b62f1ed74d079638bb32ff3079dad5b01c75" Oct 30 01:36:45.737807 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:45.737783 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-97c137-7c4fbd7b9b-rh7wf" event={"ID":"6d161227-4d8e-49b1-b4e3-308d91f85588","Type":"ContainerDied","Data":"ef36a3530f47557730a95962035308133d83530667e6f9a11510828901749405"} Oct 30 01:36:45.737807 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:45.737804 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-97c137-7c4fbd7b9b-rh7wf" Oct 30 01:36:45.746124 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:45.746107 6221 scope.go:122] "RemoveContainer" containerID="4966adbefe210ada5c23be855118b62f1ed74d079638bb32ff3079dad5b01c75" Oct 30 01:36:45.746773 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:36:45.746752 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4966adbefe210ada5c23be855118b62f1ed74d079638bb32ff3079dad5b01c75\": container with ID starting with 4966adbefe210ada5c23be855118b62f1ed74d079638bb32ff3079dad5b01c75 not found: ID does not exist" containerID="4966adbefe210ada5c23be855118b62f1ed74d079638bb32ff3079dad5b01c75" Oct 30 01:36:45.746855 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:45.746780 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4966adbefe210ada5c23be855118b62f1ed74d079638bb32ff3079dad5b01c75"} err="failed to get container status \"4966adbefe210ada5c23be855118b62f1ed74d079638bb32ff3079dad5b01c75\": rpc error: code = NotFound desc = could not find container \"4966adbefe210ada5c23be855118b62f1ed74d079638bb32ff3079dad5b01c75\": container with ID starting with 4966adbefe210ada5c23be855118b62f1ed74d079638bb32ff3079dad5b01c75 not found: ID does not exist" Oct 30 01:36:45.746855 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:45.746796 6221 scope.go:122] "RemoveContainer" containerID="3faebd6e277c3480b4230051f4cbf49d8a48b1cb42fb4975b51ecf7af0f6fc9d" Oct 30 01:36:45.789404 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:45.789352 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-97c137-7c4fbd7b9b-rh7wf"] Oct 30 01:36:45.801940 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:45.801914 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-97c137-7c4fbd7b9b-rh7wf"] Oct 30 01:36:45.827364 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:45.827328 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-97c137-64d87865f-d68pz"] Oct 30 01:36:45.830716 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:45.830692 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-97c137-64d87865f-d68pz"] Oct 30 01:36:46.548639 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:46.548607 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d161227-4d8e-49b1-b4e3-308d91f85588" path="/var/lib/kubelet/pods/6d161227-4d8e-49b1-b4e3-308d91f85588/volumes" Oct 30 01:36:46.549090 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:46.549076 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d280af82-ee8f-4f27-8cbb-2e7fbd80b5d1" path="/var/lib/kubelet/pods/d280af82-ee8f-4f27-8cbb-2e7fbd80b5d1/volumes" Oct 30 01:36:48.624839 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:36:48.624813 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f09934-98df65955-8cwfc" Oct 30 01:36:51.494295 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:36:51.494258 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:37:13.993578 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:13.993541 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf"] Oct 30 01:37:14.029206 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:14.029146 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" podUID="d30042d8-3e81-4825-9afa-9848e0f06141" containerName="istio-proxy" containerID="cri-o://c5d2697229be541f56891b6722896fcfe36f617fd174f1fdcffd32a850df5bc8" gracePeriod=30 Oct 30 01:37:15.860812 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:15.860776 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-18560b-7b94fb568-c977d"] Oct 30 01:37:15.861385 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:15.861371 6221 cpu_manager.go:414] "containerMap: removing container" podUID="d280af82-ee8f-4f27-8cbb-2e7fbd80b5d1" containerName="payload-pre-processing" Oct 30 01:37:15.861813 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:15.861386 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d280af82-ee8f-4f27-8cbb-2e7fbd80b5d1" containerName="payload-pre-processing" Oct 30 01:37:15.861813 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:15.861399 6221 cpu_manager.go:414] "containerMap: removing container" podUID="03e0644c-f8df-4da7-9414-ff2bb32e2855" containerName="main" Oct 30 01:37:15.861813 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:15.861404 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03e0644c-f8df-4da7-9414-ff2bb32e2855" containerName="main" Oct 30 01:37:15.861813 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:15.861425 6221 cpu_manager.go:414] "containerMap: removing container" podUID="0c793208-af28-4c73-a6de-9ea3f69e80d6" containerName="payload-pre-processing" Oct 30 01:37:15.861813 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:15.861433 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c793208-af28-4c73-a6de-9ea3f69e80d6" containerName="payload-pre-processing" Oct 30 01:37:15.861813 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:15.861442 6221 cpu_manager.go:414] "containerMap: removing container" podUID="6d161227-4d8e-49b1-b4e3-308d91f85588" containerName="payload-processing" Oct 30 01:37:15.861813 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:15.861446 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d161227-4d8e-49b1-b4e3-308d91f85588" containerName="payload-processing" Oct 30 01:37:15.861813 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:15.861550 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="d280af82-ee8f-4f27-8cbb-2e7fbd80b5d1" containerName="payload-pre-processing" Oct 30 01:37:15.861813 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:15.861559 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="03e0644c-f8df-4da7-9414-ff2bb32e2855" containerName="main" Oct 30 01:37:15.861813 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:15.861567 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c793208-af28-4c73-a6de-9ea3f69e80d6" containerName="payload-pre-processing" Oct 30 01:37:15.861813 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:15.861574 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d161227-4d8e-49b1-b4e3-308d91f85588" containerName="payload-processing" Oct 30 01:37:15.863876 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:15.863860 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-18560b-7b94fb568-c977d" Oct 30 01:37:15.874929 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:15.874902 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtsgv\" (UniqueName: \"kubernetes.io/projected/829e28c7-ba53-4e86-afbd-f033698c333d-kube-api-access-gtsgv\") pod \"payload-processing-e2e-api-b-18560b-7b94fb568-c977d\" (UID: \"829e28c7-ba53-4e86-afbd-f033698c333d\") " pod="openshift-ingress/payload-processing-e2e-api-b-18560b-7b94fb568-c977d" Oct 30 01:37:15.875082 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:15.875052 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/829e28c7-ba53-4e86-afbd-f033698c333d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-18560b-7b94fb568-c977d\" (UID: \"829e28c7-ba53-4e86-afbd-f033698c333d\") " pod="openshift-ingress/payload-processing-e2e-api-b-18560b-7b94fb568-c977d" Oct 30 01:37:15.882282 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:15.882258 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-18560b-7b94fb568-c977d"] Oct 30 01:37:15.890923 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:15.890900 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-18560b\"" Oct 30 01:37:15.984659 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:15.978229 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtsgv\" (UniqueName: \"kubernetes.io/projected/829e28c7-ba53-4e86-afbd-f033698c333d-kube-api-access-gtsgv\") pod \"payload-processing-e2e-api-b-18560b-7b94fb568-c977d\" (UID: \"829e28c7-ba53-4e86-afbd-f033698c333d\") " pod="openshift-ingress/payload-processing-e2e-api-b-18560b-7b94fb568-c977d" Oct 30 01:37:15.984659 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:15.978377 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/829e28c7-ba53-4e86-afbd-f033698c333d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-18560b-7b94fb568-c977d\" (UID: \"829e28c7-ba53-4e86-afbd-f033698c333d\") " pod="openshift-ingress/payload-processing-e2e-api-b-18560b-7b94fb568-c977d" Oct 30 01:37:15.984659 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:15.983128 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/829e28c7-ba53-4e86-afbd-f033698c333d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-18560b-7b94fb568-c977d\" (UID: \"829e28c7-ba53-4e86-afbd-f033698c333d\") " pod="openshift-ingress/payload-processing-e2e-api-b-18560b-7b94fb568-c977d" Oct 30 01:37:16.186599 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:16.180367 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtsgv\" (UniqueName: \"kubernetes.io/projected/829e28c7-ba53-4e86-afbd-f033698c333d-kube-api-access-gtsgv\") pod \"payload-processing-e2e-api-b-18560b-7b94fb568-c977d\" (UID: \"829e28c7-ba53-4e86-afbd-f033698c333d\") " pod="openshift-ingress/payload-processing-e2e-api-b-18560b-7b94fb568-c977d" Oct 30 01:37:16.477170 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:16.477089 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-18560b-7b94fb568-c977d" Oct 30 01:37:16.776992 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:16.776969 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-18560b-7b94fb568-c977d"] Oct 30 01:37:16.779484 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:37:16.779455 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod829e28c7_ba53_4e86_afbd_f033698c333d.slice/crio-169aa338dbb2db81fa71eed1507eb8087be97ec31872d756c5c0b10882e94215 WatchSource:0}: Error finding container 169aa338dbb2db81fa71eed1507eb8087be97ec31872d756c5c0b10882e94215: Status 404 returned error can't find the container with id 169aa338dbb2db81fa71eed1507eb8087be97ec31872d756c5c0b10882e94215 Oct 30 01:37:17.260031 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:17.259997 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-18560b-7b94fb568-c977d" event={"ID":"829e28c7-ba53-4e86-afbd-f033698c333d","Type":"ContainerStarted","Data":"8bf31446c67eb0460711618f0a35d5996cb9629b489e0c954f8acb8906fcabd0"} Oct 30 01:37:17.260525 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:17.260048 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-18560b-7b94fb568-c977d" event={"ID":"829e28c7-ba53-4e86-afbd-f033698c333d","Type":"ContainerStarted","Data":"169aa338dbb2db81fa71eed1507eb8087be97ec31872d756c5c0b10882e94215"} Oct 30 01:37:17.260525 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:17.260274 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-18560b-7b94fb568-c977d" Oct 30 01:37:17.303614 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:17.303568 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-18560b-7b94fb568-c977d" podStartSLOduration=2.303555064 podStartE2EDuration="2.303555064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:37:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:37:17.300427205 +0000 UTC m=+1160.878714540" watchObservedRunningTime="2026-10-30 01:37:17.303555064 +0000 UTC m=+1160.881842400" Oct 30 01:37:18.464051 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:18.458151 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm"] Oct 30 01:37:18.464051 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:18.458386 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" podUID="60112bce-f58b-467c-8afe-11a1bb92f891" containerName="istio-proxy" containerID="cri-o://b5cb2b99cd2d8726ea07b858d76aa93207c7edf9295615e5d411469b5cf8097a" gracePeriod=30 Oct 30 01:37:19.274424 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:19.274387 6221 generic.go:356] "Generic (PLEG): container finished" podID="d30042d8-3e81-4825-9afa-9848e0f06141" containerID="c5d2697229be541f56891b6722896fcfe36f617fd174f1fdcffd32a850df5bc8" exitCode=0 Oct 30 01:37:19.274649 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:19.274434 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" event={"ID":"d30042d8-3e81-4825-9afa-9848e0f06141","Type":"ContainerDied","Data":"c5d2697229be541f56891b6722896fcfe36f617fd174f1fdcffd32a850df5bc8"} Oct 30 01:37:19.274649 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:19.274492 6221 scope.go:122] "RemoveContainer" containerID="3249149ef74bd2d4da3f251e78605238a18942e5d5a4982ee8bc30f368553b2d" Oct 30 01:37:19.699478 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:19.699453 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" Oct 30 01:37:19.748489 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:19.748456 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d30042d8-3e81-4825-9afa-9848e0f06141-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30042d8-3e81-4825-9afa-9848e0f06141-credential-socket\") pod \"d30042d8-3e81-4825-9afa-9848e0f06141\" (UID: \"d30042d8-3e81-4825-9afa-9848e0f06141\") " Oct 30 01:37:19.748681 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:19.748500 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d30042d8-3e81-4825-9afa-9848e0f06141-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d30042d8-3e81-4825-9afa-9848e0f06141-istio-data\") pod \"d30042d8-3e81-4825-9afa-9848e0f06141\" (UID: \"d30042d8-3e81-4825-9afa-9848e0f06141\") " Oct 30 01:37:19.748681 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:19.748584 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d30042d8-3e81-4825-9afa-9848e0f06141-kube-api-access-jbrbq\" (UniqueName: \"kubernetes.io/projected/d30042d8-3e81-4825-9afa-9848e0f06141-kube-api-access-jbrbq\") pod \"d30042d8-3e81-4825-9afa-9848e0f06141\" (UID: \"d30042d8-3e81-4825-9afa-9848e0f06141\") " Oct 30 01:37:19.748681 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:19.748614 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d30042d8-3e81-4825-9afa-9848e0f06141-istio-token\" (UniqueName: \"kubernetes.io/projected/d30042d8-3e81-4825-9afa-9848e0f06141-istio-token\") pod \"d30042d8-3e81-4825-9afa-9848e0f06141\" (UID: \"d30042d8-3e81-4825-9afa-9848e0f06141\") " Oct 30 01:37:19.748844 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:19.748727 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d30042d8-3e81-4825-9afa-9848e0f06141-credential-socket" pod "d30042d8-3e81-4825-9afa-9848e0f06141" (UID: "d30042d8-3e81-4825-9afa-9848e0f06141"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:37:19.748844 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:19.748736 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d30042d8-3e81-4825-9afa-9848e0f06141-istio-data" pod "d30042d8-3e81-4825-9afa-9848e0f06141" (UID: "d30042d8-3e81-4825-9afa-9848e0f06141"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:37:19.748844 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:19.748785 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d30042d8-3e81-4825-9afa-9848e0f06141-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d30042d8-3e81-4825-9afa-9848e0f06141-istio-podinfo\") pod \"d30042d8-3e81-4825-9afa-9848e0f06141\" (UID: \"d30042d8-3e81-4825-9afa-9848e0f06141\") " Oct 30 01:37:19.748844 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:19.748814 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d30042d8-3e81-4825-9afa-9848e0f06141-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d30042d8-3e81-4825-9afa-9848e0f06141-istiod-ca-cert\") pod \"d30042d8-3e81-4825-9afa-9848e0f06141\" (UID: \"d30042d8-3e81-4825-9afa-9848e0f06141\") " Oct 30 01:37:19.749098 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:19.748850 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d30042d8-3e81-4825-9afa-9848e0f06141-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d30042d8-3e81-4825-9afa-9848e0f06141-istio-envoy\") pod \"d30042d8-3e81-4825-9afa-9848e0f06141\" (UID: \"d30042d8-3e81-4825-9afa-9848e0f06141\") " Oct 30 01:37:19.749098 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:19.748886 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d30042d8-3e81-4825-9afa-9848e0f06141-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30042d8-3e81-4825-9afa-9848e0f06141-workload-socket\") pod \"d30042d8-3e81-4825-9afa-9848e0f06141\" (UID: \"d30042d8-3e81-4825-9afa-9848e0f06141\") " Oct 30 01:37:19.749098 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:19.748955 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d30042d8-3e81-4825-9afa-9848e0f06141-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d30042d8-3e81-4825-9afa-9848e0f06141-workload-certs\") pod \"d30042d8-3e81-4825-9afa-9848e0f06141\" (UID: \"d30042d8-3e81-4825-9afa-9848e0f06141\") " Oct 30 01:37:19.749255 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:19.749233 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d30042d8-3e81-4825-9afa-9848e0f06141-istiod-ca-cert" pod "d30042d8-3e81-4825-9afa-9848e0f06141" (UID: "d30042d8-3e81-4825-9afa-9848e0f06141"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:37:19.749552 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:19.749506 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d30042d8-3e81-4825-9afa-9848e0f06141-workload-socket" pod "d30042d8-3e81-4825-9afa-9848e0f06141" (UID: "d30042d8-3e81-4825-9afa-9848e0f06141"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:37:19.749654 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:19.749583 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30042d8-3e81-4825-9afa-9848e0f06141-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:37:19.749654 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:19.749605 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d30042d8-3e81-4825-9afa-9848e0f06141-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:37:19.749654 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:19.749620 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d30042d8-3e81-4825-9afa-9848e0f06141-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:37:19.749654 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:19.749642 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d30042d8-3e81-4825-9afa-9848e0f06141-workload-certs" pod "d30042d8-3e81-4825-9afa-9848e0f06141" (UID: "d30042d8-3e81-4825-9afa-9848e0f06141"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:37:19.751095 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:19.751066 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d30042d8-3e81-4825-9afa-9848e0f06141-kube-api-access-jbrbq" pod "d30042d8-3e81-4825-9afa-9848e0f06141" (UID: "d30042d8-3e81-4825-9afa-9848e0f06141"). InnerVolumeSpecName "kube-api-access-jbrbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:37:19.751333 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:19.751310 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d30042d8-3e81-4825-9afa-9848e0f06141-istio-token" pod "d30042d8-3e81-4825-9afa-9848e0f06141" (UID: "d30042d8-3e81-4825-9afa-9848e0f06141"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:37:19.751480 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:19.751463 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d30042d8-3e81-4825-9afa-9848e0f06141-istio-envoy" pod "d30042d8-3e81-4825-9afa-9848e0f06141" (UID: "d30042d8-3e81-4825-9afa-9848e0f06141"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:37:19.751625 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:19.751609 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d30042d8-3e81-4825-9afa-9848e0f06141-istio-podinfo" pod "d30042d8-3e81-4825-9afa-9848e0f06141" (UID: "d30042d8-3e81-4825-9afa-9848e0f06141"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:37:19.851572 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:19.851476 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d30042d8-3e81-4825-9afa-9848e0f06141-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:37:19.851572 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:19.851506 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d30042d8-3e81-4825-9afa-9848e0f06141-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:37:19.851572 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:19.851520 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d30042d8-3e81-4825-9afa-9848e0f06141-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:37:19.851572 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:19.851564 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d30042d8-3e81-4825-9afa-9848e0f06141-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:37:19.851821 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:19.851579 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbrbq\" (UniqueName: \"kubernetes.io/projected/d30042d8-3e81-4825-9afa-9848e0f06141-kube-api-access-jbrbq\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:37:19.851821 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:19.851595 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d30042d8-3e81-4825-9afa-9848e0f06141-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:37:20.282839 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:20.282812 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" Oct 30 01:37:20.283013 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:20.282840 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf" event={"ID":"d30042d8-3e81-4825-9afa-9848e0f06141","Type":"ContainerDied","Data":"7f057a72629b8834b12e3b49dee6bdcb74da2312efbb44e50aaf0f3489c141c6"} Oct 30 01:37:20.283013 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:20.282921 6221 scope.go:122] "RemoveContainer" containerID="c5d2697229be541f56891b6722896fcfe36f617fd174f1fdcffd32a850df5bc8" Oct 30 01:37:20.348315 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:20.348284 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf"] Oct 30 01:37:20.360380 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:20.360352 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-97c137-openshift-default-6d4c6b8fd7-bwrgf"] Oct 30 01:37:20.548153 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:20.548076 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d30042d8-3e81-4825-9afa-9848e0f06141" path="/var/lib/kubelet/pods/d30042d8-3e81-4825-9afa-9848e0f06141/volumes" Oct 30 01:37:21.494769 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:37:21.494732 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:37:22.938566 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:22.937797 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:37:23.402825 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:23.402789 6221 patch_prober.go:29] interesting pod/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:37:23.402973 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:23.402845 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" podUID="60112bce-f58b-467c-8afe-11a1bb92f891" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:37:23.725919 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:23.725894 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" Oct 30 01:37:23.829109 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:23.829069 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60112bce-f58b-467c-8afe-11a1bb92f891-kube-api-access-4jv8v\" (UniqueName: \"kubernetes.io/projected/60112bce-f58b-467c-8afe-11a1bb92f891-kube-api-access-4jv8v\") pod \"60112bce-f58b-467c-8afe-11a1bb92f891\" (UID: \"60112bce-f58b-467c-8afe-11a1bb92f891\") " Oct 30 01:37:23.829838 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:23.829814 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/60112bce-f58b-467c-8afe-11a1bb92f891-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60112bce-f58b-467c-8afe-11a1bb92f891-istio-podinfo\") pod \"60112bce-f58b-467c-8afe-11a1bb92f891\" (UID: \"60112bce-f58b-467c-8afe-11a1bb92f891\") " Oct 30 01:37:23.830452 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:23.830434 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60112bce-f58b-467c-8afe-11a1bb92f891-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60112bce-f58b-467c-8afe-11a1bb92f891-istio-data\") pod \"60112bce-f58b-467c-8afe-11a1bb92f891\" (UID: \"60112bce-f58b-467c-8afe-11a1bb92f891\") " Oct 30 01:37:23.831215 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:23.830818 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60112bce-f58b-467c-8afe-11a1bb92f891-istio-data" pod "60112bce-f58b-467c-8afe-11a1bb92f891" (UID: "60112bce-f58b-467c-8afe-11a1bb92f891"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:37:23.831573 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:23.831555 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60112bce-f58b-467c-8afe-11a1bb92f891-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60112bce-f58b-467c-8afe-11a1bb92f891-workload-certs\") pod \"60112bce-f58b-467c-8afe-11a1bb92f891\" (UID: \"60112bce-f58b-467c-8afe-11a1bb92f891\") " Oct 30 01:37:23.832033 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:23.832013 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60112bce-f58b-467c-8afe-11a1bb92f891-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60112bce-f58b-467c-8afe-11a1bb92f891-workload-socket\") pod \"60112bce-f58b-467c-8afe-11a1bb92f891\" (UID: \"60112bce-f58b-467c-8afe-11a1bb92f891\") " Oct 30 01:37:23.832423 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:23.832404 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60112bce-f58b-467c-8afe-11a1bb92f891-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60112bce-f58b-467c-8afe-11a1bb92f891-credential-socket\") pod \"60112bce-f58b-467c-8afe-11a1bb92f891\" (UID: \"60112bce-f58b-467c-8afe-11a1bb92f891\") " Oct 30 01:37:23.832851 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:23.832833 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60112bce-f58b-467c-8afe-11a1bb92f891-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60112bce-f58b-467c-8afe-11a1bb92f891-istiod-ca-cert\") pod \"60112bce-f58b-467c-8afe-11a1bb92f891\" (UID: \"60112bce-f58b-467c-8afe-11a1bb92f891\") " Oct 30 01:37:23.833291 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:23.833275 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60112bce-f58b-467c-8afe-11a1bb92f891-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60112bce-f58b-467c-8afe-11a1bb92f891-istio-envoy\") pod \"60112bce-f58b-467c-8afe-11a1bb92f891\" (UID: \"60112bce-f58b-467c-8afe-11a1bb92f891\") " Oct 30 01:37:23.833736 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:23.833719 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60112bce-f58b-467c-8afe-11a1bb92f891-istio-token\" (UniqueName: \"kubernetes.io/projected/60112bce-f58b-467c-8afe-11a1bb92f891-istio-token\") pod \"60112bce-f58b-467c-8afe-11a1bb92f891\" (UID: \"60112bce-f58b-467c-8afe-11a1bb92f891\") " Oct 30 01:37:23.834926 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:23.834905 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60112bce-f58b-467c-8afe-11a1bb92f891-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:37:23.835700 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:23.831934 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60112bce-f58b-467c-8afe-11a1bb92f891-workload-certs" pod "60112bce-f58b-467c-8afe-11a1bb92f891" (UID: "60112bce-f58b-467c-8afe-11a1bb92f891"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:37:23.835823 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:23.832322 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60112bce-f58b-467c-8afe-11a1bb92f891-workload-socket" pod "60112bce-f58b-467c-8afe-11a1bb92f891" (UID: "60112bce-f58b-467c-8afe-11a1bb92f891"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:37:23.835905 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:23.832731 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60112bce-f58b-467c-8afe-11a1bb92f891-credential-socket" pod "60112bce-f58b-467c-8afe-11a1bb92f891" (UID: "60112bce-f58b-467c-8afe-11a1bb92f891"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:37:23.835977 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:23.833226 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60112bce-f58b-467c-8afe-11a1bb92f891-istiod-ca-cert" pod "60112bce-f58b-467c-8afe-11a1bb92f891" (UID: "60112bce-f58b-467c-8afe-11a1bb92f891"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:37:23.845905 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:23.845854 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60112bce-f58b-467c-8afe-11a1bb92f891-istio-token" pod "60112bce-f58b-467c-8afe-11a1bb92f891" (UID: "60112bce-f58b-467c-8afe-11a1bb92f891"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:37:23.846515 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:23.846491 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60112bce-f58b-467c-8afe-11a1bb92f891-istio-podinfo" pod "60112bce-f58b-467c-8afe-11a1bb92f891" (UID: "60112bce-f58b-467c-8afe-11a1bb92f891"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:37:23.846664 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:23.846638 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60112bce-f58b-467c-8afe-11a1bb92f891-istio-envoy" pod "60112bce-f58b-467c-8afe-11a1bb92f891" (UID: "60112bce-f58b-467c-8afe-11a1bb92f891"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:37:23.848193 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:23.848162 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60112bce-f58b-467c-8afe-11a1bb92f891-kube-api-access-4jv8v" pod "60112bce-f58b-467c-8afe-11a1bb92f891" (UID: "60112bce-f58b-467c-8afe-11a1bb92f891"). InnerVolumeSpecName "kube-api-access-4jv8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:37:23.971637 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:23.968322 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60112bce-f58b-467c-8afe-11a1bb92f891-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:37:23.971637 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:23.968372 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60112bce-f58b-467c-8afe-11a1bb92f891-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:37:23.971637 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:23.968385 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60112bce-f58b-467c-8afe-11a1bb92f891-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:37:23.971637 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:23.968399 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60112bce-f58b-467c-8afe-11a1bb92f891-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:37:23.971637 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:23.968444 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60112bce-f58b-467c-8afe-11a1bb92f891-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:37:23.971637 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:23.968458 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60112bce-f58b-467c-8afe-11a1bb92f891-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:37:23.971637 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:23.968472 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jv8v\" (UniqueName: \"kubernetes.io/projected/60112bce-f58b-467c-8afe-11a1bb92f891-kube-api-access-4jv8v\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:37:23.971637 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:23.968488 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60112bce-f58b-467c-8afe-11a1bb92f891-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:37:24.313177 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:24.313140 6221 generic.go:356] "Generic (PLEG): container finished" podID="60112bce-f58b-467c-8afe-11a1bb92f891" containerID="b5cb2b99cd2d8726ea07b858d76aa93207c7edf9295615e5d411469b5cf8097a" exitCode=0 Oct 30 01:37:24.313372 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:24.313228 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" Oct 30 01:37:24.313470 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:24.313225 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" event={"ID":"60112bce-f58b-467c-8afe-11a1bb92f891","Type":"ContainerDied","Data":"b5cb2b99cd2d8726ea07b858d76aa93207c7edf9295615e5d411469b5cf8097a"} Oct 30 01:37:24.313554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:24.313474 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm" event={"ID":"60112bce-f58b-467c-8afe-11a1bb92f891","Type":"ContainerDied","Data":"2e4bbf9f532372379e3a9fcdd6e801369703908f1f06bec44e9291c4c2c545c8"} Oct 30 01:37:24.313554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:24.313513 6221 scope.go:122] "RemoveContainer" containerID="b5cb2b99cd2d8726ea07b858d76aa93207c7edf9295615e5d411469b5cf8097a" Oct 30 01:37:24.328165 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:24.328137 6221 scope.go:122] "RemoveContainer" containerID="13cfd9a06c7ca72616f6a22f8316aa683814cc70af429cc4e1542bae0d310df5" Oct 30 01:37:24.340983 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:24.340949 6221 scope.go:122] "RemoveContainer" containerID="b5cb2b99cd2d8726ea07b858d76aa93207c7edf9295615e5d411469b5cf8097a" Oct 30 01:37:24.341778 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:37:24.341746 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5cb2b99cd2d8726ea07b858d76aa93207c7edf9295615e5d411469b5cf8097a\": container with ID starting with b5cb2b99cd2d8726ea07b858d76aa93207c7edf9295615e5d411469b5cf8097a not found: ID does not exist" containerID="b5cb2b99cd2d8726ea07b858d76aa93207c7edf9295615e5d411469b5cf8097a" Oct 30 01:37:24.341891 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:24.341787 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5cb2b99cd2d8726ea07b858d76aa93207c7edf9295615e5d411469b5cf8097a"} err="failed to get container status \"b5cb2b99cd2d8726ea07b858d76aa93207c7edf9295615e5d411469b5cf8097a\": rpc error: code = NotFound desc = could not find container \"b5cb2b99cd2d8726ea07b858d76aa93207c7edf9295615e5d411469b5cf8097a\": container with ID starting with b5cb2b99cd2d8726ea07b858d76aa93207c7edf9295615e5d411469b5cf8097a not found: ID does not exist" Oct 30 01:37:24.341891 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:24.341810 6221 scope.go:122] "RemoveContainer" containerID="13cfd9a06c7ca72616f6a22f8316aa683814cc70af429cc4e1542bae0d310df5" Oct 30 01:37:24.342141 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:37:24.342116 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13cfd9a06c7ca72616f6a22f8316aa683814cc70af429cc4e1542bae0d310df5\": container with ID starting with 13cfd9a06c7ca72616f6a22f8316aa683814cc70af429cc4e1542bae0d310df5 not found: ID does not exist" containerID="13cfd9a06c7ca72616f6a22f8316aa683814cc70af429cc4e1542bae0d310df5" Oct 30 01:37:24.342237 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:24.342149 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13cfd9a06c7ca72616f6a22f8316aa683814cc70af429cc4e1542bae0d310df5"} err="failed to get container status \"13cfd9a06c7ca72616f6a22f8316aa683814cc70af429cc4e1542bae0d310df5\": rpc error: code = NotFound desc = could not find container \"13cfd9a06c7ca72616f6a22f8316aa683814cc70af429cc4e1542bae0d310df5\": container with ID starting with 13cfd9a06c7ca72616f6a22f8316aa683814cc70af429cc4e1542bae0d310df5 not found: ID does not exist" Oct 30 01:37:24.381074 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:24.381041 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm"] Oct 30 01:37:24.393806 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:24.393777 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-288c8f-openshift-default-5766785b8c-nrhqm"] Oct 30 01:37:24.553369 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:24.553322 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60112bce-f58b-467c-8afe-11a1bb92f891" path="/var/lib/kubelet/pods/60112bce-f58b-467c-8afe-11a1bb92f891/volumes" Oct 30 01:37:26.379557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:26.374650 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-wvm5b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:37:26.379557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:26.374728 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b" podUID="f33783d9-0aa2-4089-878d-ffc93fded2c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:37:28.269282 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:28.269257 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-18560b-7b94fb568-c977d" Oct 30 01:37:28.364604 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:28.364573 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-18560b-698849667d-6tdwg"] Oct 30 01:37:28.365185 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:28.365171 6221 cpu_manager.go:414] "containerMap: removing container" podUID="d30042d8-3e81-4825-9afa-9848e0f06141" containerName="istio-proxy" Oct 30 01:37:28.365237 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:28.365187 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d30042d8-3e81-4825-9afa-9848e0f06141" containerName="istio-proxy" Oct 30 01:37:28.365237 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:28.365206 6221 cpu_manager.go:414] "containerMap: removing container" podUID="60112bce-f58b-467c-8afe-11a1bb92f891" containerName="istio-proxy" Oct 30 01:37:28.365237 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:28.365213 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60112bce-f58b-467c-8afe-11a1bb92f891" containerName="istio-proxy" Oct 30 01:37:28.365237 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:28.365236 6221 cpu_manager.go:414] "containerMap: removing container" podUID="60112bce-f58b-467c-8afe-11a1bb92f891" containerName="istio-proxy" Oct 30 01:37:28.365361 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:28.365241 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60112bce-f58b-467c-8afe-11a1bb92f891" containerName="istio-proxy" Oct 30 01:37:28.365361 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:28.365341 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="d30042d8-3e81-4825-9afa-9848e0f06141" containerName="istio-proxy" Oct 30 01:37:28.365361 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:28.365351 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="60112bce-f58b-467c-8afe-11a1bb92f891" containerName="istio-proxy" Oct 30 01:37:28.365463 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:28.365365 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="d30042d8-3e81-4825-9afa-9848e0f06141" containerName="istio-proxy" Oct 30 01:37:28.365463 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:28.365372 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="60112bce-f58b-467c-8afe-11a1bb92f891" containerName="istio-proxy" Oct 30 01:37:28.368200 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:28.368180 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18560b-698849667d-6tdwg" Oct 30 01:37:28.388846 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:28.388821 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-18560b-serving-cert\"" Oct 30 01:37:28.394964 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:28.393366 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-18560b-698849667d-6tdwg"] Oct 30 01:37:28.498053 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:28.498020 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf83d1bd-9ffa-4367-b8ce-80978585a9f2-openshift-service-ca\") pod \"maas-api-e2e-api-b-18560b-698849667d-6tdwg\" (UID: \"cf83d1bd-9ffa-4367-b8ce-80978585a9f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18560b-698849667d-6tdwg" Oct 30 01:37:28.498053 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:28.498052 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf83d1bd-9ffa-4367-b8ce-80978585a9f2-metrics-certs\") pod \"maas-api-e2e-api-b-18560b-698849667d-6tdwg\" (UID: \"cf83d1bd-9ffa-4367-b8ce-80978585a9f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18560b-698849667d-6tdwg" Oct 30 01:37:28.498256 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:28.498144 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqpxx\" (UniqueName: \"kubernetes.io/projected/cf83d1bd-9ffa-4367-b8ce-80978585a9f2-kube-api-access-kqpxx\") pod \"maas-api-e2e-api-b-18560b-698849667d-6tdwg\" (UID: \"cf83d1bd-9ffa-4367-b8ce-80978585a9f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18560b-698849667d-6tdwg" Oct 30 01:37:28.498338 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:28.498319 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf83d1bd-9ffa-4367-b8ce-80978585a9f2-maas-api-tls\") pod \"maas-api-e2e-api-b-18560b-698849667d-6tdwg\" (UID: \"cf83d1bd-9ffa-4367-b8ce-80978585a9f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18560b-698849667d-6tdwg" Oct 30 01:37:28.605517 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:28.605478 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqpxx\" (UniqueName: \"kubernetes.io/projected/cf83d1bd-9ffa-4367-b8ce-80978585a9f2-kube-api-access-kqpxx\") pod \"maas-api-e2e-api-b-18560b-698849667d-6tdwg\" (UID: \"cf83d1bd-9ffa-4367-b8ce-80978585a9f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18560b-698849667d-6tdwg" Oct 30 01:37:28.606030 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:28.606005 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf83d1bd-9ffa-4367-b8ce-80978585a9f2-maas-api-tls\") pod \"maas-api-e2e-api-b-18560b-698849667d-6tdwg\" (UID: \"cf83d1bd-9ffa-4367-b8ce-80978585a9f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18560b-698849667d-6tdwg" Oct 30 01:37:28.607494 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:28.607467 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf83d1bd-9ffa-4367-b8ce-80978585a9f2-openshift-service-ca\") pod \"maas-api-e2e-api-b-18560b-698849667d-6tdwg\" (UID: \"cf83d1bd-9ffa-4367-b8ce-80978585a9f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18560b-698849667d-6tdwg" Oct 30 01:37:28.607703 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:28.607682 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf83d1bd-9ffa-4367-b8ce-80978585a9f2-metrics-certs\") pod \"maas-api-e2e-api-b-18560b-698849667d-6tdwg\" (UID: \"cf83d1bd-9ffa-4367-b8ce-80978585a9f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18560b-698849667d-6tdwg" Oct 30 01:37:28.609309 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:28.609284 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf83d1bd-9ffa-4367-b8ce-80978585a9f2-openshift-service-ca\") pod \"maas-api-e2e-api-b-18560b-698849667d-6tdwg\" (UID: \"cf83d1bd-9ffa-4367-b8ce-80978585a9f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18560b-698849667d-6tdwg" Oct 30 01:37:28.620283 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:28.620251 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf83d1bd-9ffa-4367-b8ce-80978585a9f2-metrics-certs\") pod \"maas-api-e2e-api-b-18560b-698849667d-6tdwg\" (UID: \"cf83d1bd-9ffa-4367-b8ce-80978585a9f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18560b-698849667d-6tdwg" Oct 30 01:37:28.649652 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:28.648074 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf83d1bd-9ffa-4367-b8ce-80978585a9f2-maas-api-tls\") pod \"maas-api-e2e-api-b-18560b-698849667d-6tdwg\" (UID: \"cf83d1bd-9ffa-4367-b8ce-80978585a9f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18560b-698849667d-6tdwg" Oct 30 01:37:28.657429 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:28.657388 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqpxx\" (UniqueName: \"kubernetes.io/projected/cf83d1bd-9ffa-4367-b8ce-80978585a9f2-kube-api-access-kqpxx\") pod \"maas-api-e2e-api-b-18560b-698849667d-6tdwg\" (UID: \"cf83d1bd-9ffa-4367-b8ce-80978585a9f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18560b-698849667d-6tdwg" Oct 30 01:37:28.684576 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:28.683455 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18560b-698849667d-6tdwg" Oct 30 01:37:29.029277 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:29.029253 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-18560b-698849667d-6tdwg"] Oct 30 01:37:29.031790 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:37:29.031764 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf83d1bd_9ffa_4367_b8ce_80978585a9f2.slice/crio-92250ca3386f379fa02474dca24d735d9a98a5c5027755a9bcbf657d1429926c WatchSource:0}: Error finding container 92250ca3386f379fa02474dca24d735d9a98a5c5027755a9bcbf657d1429926c: Status 404 returned error can't find the container with id 92250ca3386f379fa02474dca24d735d9a98a5c5027755a9bcbf657d1429926c Oct 30 01:37:29.406772 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:29.406741 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18560b-698849667d-6tdwg" event={"ID":"cf83d1bd-9ffa-4367-b8ce-80978585a9f2","Type":"ContainerStarted","Data":"92250ca3386f379fa02474dca24d735d9a98a5c5027755a9bcbf657d1429926c"} Oct 30 01:37:30.444556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:30.439600 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18560b-698849667d-6tdwg" event={"ID":"cf83d1bd-9ffa-4367-b8ce-80978585a9f2","Type":"ContainerStarted","Data":"22a9f76deeeb7156138f0226b20bcfe3098cf5242dd0e3ea2935eba70805d0b2"} Oct 30 01:37:30.444556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:30.440784 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18560b-698849667d-6tdwg" Oct 30 01:37:30.495556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:30.494491 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18560b-698849667d-6tdwg" podStartSLOduration=2.138321288 podStartE2EDuration="2.494472172s" totalImagesPullingTime="356.150884ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:37:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:37:30.494091146 +0000 UTC m=+1174.072378475" watchObservedRunningTime="2026-10-30 01:37:30.494472172 +0000 UTC m=+1174.072759509" Oct 30 01:37:33.962288 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:33.962252 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9"] Oct 30 01:37:33.963228 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:33.963207 6221 cpu_manager.go:414] "containerMap: removing container" podUID="d30042d8-3e81-4825-9afa-9848e0f06141" containerName="istio-proxy" Oct 30 01:37:33.963285 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:33.963232 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d30042d8-3e81-4825-9afa-9848e0f06141" containerName="istio-proxy" Oct 30 01:37:33.967037 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:33.967014 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" Oct 30 01:37:33.985751 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:33.985725 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-19a5f1-openshift-default-dockercfg-kd5wl\"" Oct 30 01:37:34.002355 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:34.002324 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9"] Oct 30 01:37:34.143564 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:34.123439 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a3609a-8ccb-4149-82ec-98834c8f95a3-credential-socket\") pod \"e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9\" (UID: \"43a3609a-8ccb-4149-82ec-98834c8f95a3\") " pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" Oct 30 01:37:34.144236 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:34.144183 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43a3609a-8ccb-4149-82ec-98834c8f95a3-istiod-ca-cert\") pod \"e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9\" (UID: \"43a3609a-8ccb-4149-82ec-98834c8f95a3\") " pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" Oct 30 01:37:34.144504 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:34.144487 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43a3609a-8ccb-4149-82ec-98834c8f95a3-workload-certs\") pod \"e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9\" (UID: \"43a3609a-8ccb-4149-82ec-98834c8f95a3\") " pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" Oct 30 01:37:34.144734 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:34.144717 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43a3609a-8ccb-4149-82ec-98834c8f95a3-istio-data\") pod \"e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9\" (UID: \"43a3609a-8ccb-4149-82ec-98834c8f95a3\") " pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" Oct 30 01:37:34.145082 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:34.145045 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43a3609a-8ccb-4149-82ec-98834c8f95a3-istio-podinfo\") pod \"e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9\" (UID: \"43a3609a-8ccb-4149-82ec-98834c8f95a3\") " pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" Oct 30 01:37:34.145331 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:34.145306 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43a3609a-8ccb-4149-82ec-98834c8f95a3-istio-envoy\") pod \"e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9\" (UID: \"43a3609a-8ccb-4149-82ec-98834c8f95a3\") " pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" Oct 30 01:37:34.145586 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:34.145565 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8zrz\" (UniqueName: \"kubernetes.io/projected/43a3609a-8ccb-4149-82ec-98834c8f95a3-kube-api-access-j8zrz\") pod \"e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9\" (UID: \"43a3609a-8ccb-4149-82ec-98834c8f95a3\") " pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" Oct 30 01:37:34.145882 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:34.145828 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43a3609a-8ccb-4149-82ec-98834c8f95a3-istio-token\") pod \"e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9\" (UID: \"43a3609a-8ccb-4149-82ec-98834c8f95a3\") " pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" Oct 30 01:37:34.146087 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:34.146071 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a3609a-8ccb-4149-82ec-98834c8f95a3-workload-socket\") pod \"e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9\" (UID: \"43a3609a-8ccb-4149-82ec-98834c8f95a3\") " pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" Oct 30 01:37:34.259602 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:34.256574 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8zrz\" (UniqueName: \"kubernetes.io/projected/43a3609a-8ccb-4149-82ec-98834c8f95a3-kube-api-access-j8zrz\") pod \"e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9\" (UID: \"43a3609a-8ccb-4149-82ec-98834c8f95a3\") " pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" Oct 30 01:37:34.259602 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:34.256666 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43a3609a-8ccb-4149-82ec-98834c8f95a3-istio-token\") pod \"e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9\" (UID: \"43a3609a-8ccb-4149-82ec-98834c8f95a3\") " pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" Oct 30 01:37:34.259602 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:34.256723 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a3609a-8ccb-4149-82ec-98834c8f95a3-workload-socket\") pod \"e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9\" (UID: \"43a3609a-8ccb-4149-82ec-98834c8f95a3\") " pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" Oct 30 01:37:34.259602 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:34.256766 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a3609a-8ccb-4149-82ec-98834c8f95a3-credential-socket\") pod \"e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9\" (UID: \"43a3609a-8ccb-4149-82ec-98834c8f95a3\") " pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" Oct 30 01:37:34.259602 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:34.256842 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43a3609a-8ccb-4149-82ec-98834c8f95a3-istiod-ca-cert\") pod \"e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9\" (UID: \"43a3609a-8ccb-4149-82ec-98834c8f95a3\") " pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" Oct 30 01:37:34.259602 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:34.256869 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43a3609a-8ccb-4149-82ec-98834c8f95a3-workload-certs\") pod \"e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9\" (UID: \"43a3609a-8ccb-4149-82ec-98834c8f95a3\") " pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" Oct 30 01:37:34.259602 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:34.256912 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43a3609a-8ccb-4149-82ec-98834c8f95a3-istio-data\") pod \"e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9\" (UID: \"43a3609a-8ccb-4149-82ec-98834c8f95a3\") " pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" Oct 30 01:37:34.259602 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:34.257049 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43a3609a-8ccb-4149-82ec-98834c8f95a3-istio-podinfo\") pod \"e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9\" (UID: \"43a3609a-8ccb-4149-82ec-98834c8f95a3\") " pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" Oct 30 01:37:34.259602 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:34.257117 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43a3609a-8ccb-4149-82ec-98834c8f95a3-istio-envoy\") pod \"e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9\" (UID: \"43a3609a-8ccb-4149-82ec-98834c8f95a3\") " pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" Oct 30 01:37:34.259602 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:34.258502 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a3609a-8ccb-4149-82ec-98834c8f95a3-workload-socket\") pod \"e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9\" (UID: \"43a3609a-8ccb-4149-82ec-98834c8f95a3\") " pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" Oct 30 01:37:34.260619 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:34.260594 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a3609a-8ccb-4149-82ec-98834c8f95a3-credential-socket\") pod \"e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9\" (UID: \"43a3609a-8ccb-4149-82ec-98834c8f95a3\") " pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" Oct 30 01:37:34.261564 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:34.261524 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43a3609a-8ccb-4149-82ec-98834c8f95a3-istiod-ca-cert\") pod \"e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9\" (UID: \"43a3609a-8ccb-4149-82ec-98834c8f95a3\") " pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" Oct 30 01:37:34.262046 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:34.262025 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43a3609a-8ccb-4149-82ec-98834c8f95a3-workload-certs\") pod \"e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9\" (UID: \"43a3609a-8ccb-4149-82ec-98834c8f95a3\") " pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" Oct 30 01:37:34.275787 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:34.262386 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43a3609a-8ccb-4149-82ec-98834c8f95a3-istio-data\") pod \"e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9\" (UID: \"43a3609a-8ccb-4149-82ec-98834c8f95a3\") " pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" Oct 30 01:37:34.291588 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:34.287846 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43a3609a-8ccb-4149-82ec-98834c8f95a3-istio-podinfo\") pod \"e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9\" (UID: \"43a3609a-8ccb-4149-82ec-98834c8f95a3\") " pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" Oct 30 01:37:34.299568 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:34.299146 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43a3609a-8ccb-4149-82ec-98834c8f95a3-istio-envoy\") pod \"e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9\" (UID: \"43a3609a-8ccb-4149-82ec-98834c8f95a3\") " pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" Oct 30 01:37:34.299568 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:34.299439 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8zrz\" (UniqueName: \"kubernetes.io/projected/43a3609a-8ccb-4149-82ec-98834c8f95a3-kube-api-access-j8zrz\") pod \"e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9\" (UID: \"43a3609a-8ccb-4149-82ec-98834c8f95a3\") " pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" Oct 30 01:37:34.299568 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:34.299588 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43a3609a-8ccb-4149-82ec-98834c8f95a3-istio-token\") pod \"e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9\" (UID: \"43a3609a-8ccb-4149-82ec-98834c8f95a3\") " pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" Oct 30 01:37:34.588776 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:34.588742 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" Oct 30 01:37:34.909077 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:34.909050 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9"] Oct 30 01:37:34.911686 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:37:34.911661 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43a3609a_8ccb_4149_82ec_98834c8f95a3.slice/crio-b22888479b59abea944b3bf82b3efc26c514cace76a03750c2d58e18ce630875 WatchSource:0}: Error finding container b22888479b59abea944b3bf82b3efc26c514cace76a03750c2d58e18ce630875: Status 404 returned error can't find the container with id b22888479b59abea944b3bf82b3efc26c514cace76a03750c2d58e18ce630875 Oct 30 01:37:34.914393 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:34.914362 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:37:34.914488 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:34.914429 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:37:34.914488 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:34.914456 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:37:35.510621 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:35.510582 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" event={"ID":"43a3609a-8ccb-4149-82ec-98834c8f95a3","Type":"ContainerStarted","Data":"3fd81a3eef3c036d1ffa2979aeba8042424968be064164f5de527228f05d70b2"} Oct 30 01:37:35.511119 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:35.510662 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" event={"ID":"43a3609a-8ccb-4149-82ec-98834c8f95a3","Type":"ContainerStarted","Data":"b22888479b59abea944b3bf82b3efc26c514cace76a03750c2d58e18ce630875"} Oct 30 01:37:35.589128 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:35.589096 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" Oct 30 01:37:35.594972 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:35.594947 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" Oct 30 01:37:35.645520 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:35.645472 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" podStartSLOduration=2.645433938 podStartE2EDuration="2.645433938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:37:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:37:35.594826319 +0000 UTC m=+1179.173113655" watchObservedRunningTime="2026-10-30 01:37:35.645433938 +0000 UTC m=+1179.223721274" Oct 30 01:37:36.517947 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:36.517912 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" Oct 30 01:37:36.518779 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:36.518761 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" Oct 30 01:37:37.627693 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:37.621751 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18560b-698849667d-6tdwg" Oct 30 01:37:40.273830 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:40.273793 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-19a5f1-5996c85f9d-5j4nj"] Oct 30 01:37:40.277139 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:40.277116 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-19a5f1-5996c85f9d-5j4nj" Oct 30 01:37:40.296514 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:40.296487 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-19a5f1-5996c85f9d-5j4nj"] Oct 30 01:37:40.304645 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:40.304618 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-19a5f1\"" Oct 30 01:37:40.304763 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:40.304672 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-19a5f1-dockercfg-gdtlr\"" Oct 30 01:37:40.350165 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:40.350132 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15d60827-7a38-4504-9e2c-272a8ba49167-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-19a5f1-5996c85f9d-5j4nj\" (UID: \"15d60827-7a38-4504-9e2c-272a8ba49167\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-19a5f1-5996c85f9d-5j4nj" Oct 30 01:37:40.350371 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:40.350319 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt549\" (UniqueName: \"kubernetes.io/projected/15d60827-7a38-4504-9e2c-272a8ba49167-kube-api-access-mt549\") pod \"payload-processing-e2e-ipp-a-19a5f1-5996c85f9d-5j4nj\" (UID: \"15d60827-7a38-4504-9e2c-272a8ba49167\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-19a5f1-5996c85f9d-5j4nj" Oct 30 01:37:40.536616 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:40.532866 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15d60827-7a38-4504-9e2c-272a8ba49167-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-19a5f1-5996c85f9d-5j4nj\" (UID: \"15d60827-7a38-4504-9e2c-272a8ba49167\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-19a5f1-5996c85f9d-5j4nj" Oct 30 01:37:40.536616 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:40.533064 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt549\" (UniqueName: \"kubernetes.io/projected/15d60827-7a38-4504-9e2c-272a8ba49167-kube-api-access-mt549\") pod \"payload-processing-e2e-ipp-a-19a5f1-5996c85f9d-5j4nj\" (UID: \"15d60827-7a38-4504-9e2c-272a8ba49167\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-19a5f1-5996c85f9d-5j4nj" Oct 30 01:37:40.536616 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:40.534358 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15d60827-7a38-4504-9e2c-272a8ba49167-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-19a5f1-5996c85f9d-5j4nj\" (UID: \"15d60827-7a38-4504-9e2c-272a8ba49167\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-19a5f1-5996c85f9d-5j4nj" Oct 30 01:37:40.627596 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:40.622666 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt549\" (UniqueName: \"kubernetes.io/projected/15d60827-7a38-4504-9e2c-272a8ba49167-kube-api-access-mt549\") pod \"payload-processing-e2e-ipp-a-19a5f1-5996c85f9d-5j4nj\" (UID: \"15d60827-7a38-4504-9e2c-272a8ba49167\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-19a5f1-5996c85f9d-5j4nj" Oct 30 01:37:40.890709 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:40.890633 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-19a5f1-5996c85f9d-5j4nj" Oct 30 01:37:41.201465 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:41.201435 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-19a5f1-5996c85f9d-5j4nj"] Oct 30 01:37:41.203709 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:37:41.203680 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15d60827_7a38_4504_9e2c_272a8ba49167.slice/crio-d61676fe0c89b3e4ffbfce7246a59626a3da32861cdb53cbb682a0e4ceda6f57 WatchSource:0}: Error finding container d61676fe0c89b3e4ffbfce7246a59626a3da32861cdb53cbb682a0e4ceda6f57: Status 404 returned error can't find the container with id d61676fe0c89b3e4ffbfce7246a59626a3da32861cdb53cbb682a0e4ceda6f57 Oct 30 01:37:41.714120 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:41.714087 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-19a5f1-5996c85f9d-5j4nj" event={"ID":"15d60827-7a38-4504-9e2c-272a8ba49167","Type":"ContainerStarted","Data":"e89af969134d09cef09d1db3d5b8702b3f503e05142e99c3307c88e46271e826"} Oct 30 01:37:41.714517 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:41.714137 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-19a5f1-5996c85f9d-5j4nj" event={"ID":"15d60827-7a38-4504-9e2c-272a8ba49167","Type":"ContainerStarted","Data":"d61676fe0c89b3e4ffbfce7246a59626a3da32861cdb53cbb682a0e4ceda6f57"} Oct 30 01:37:41.714517 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:41.714194 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-19a5f1-5996c85f9d-5j4nj" Oct 30 01:37:41.754366 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:41.754320 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-19a5f1-5996c85f9d-5j4nj" podStartSLOduration=1.754307383 podStartE2EDuration="1.754307383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:37:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:37:41.752395326 +0000 UTC m=+1185.330682668" watchObservedRunningTime="2026-10-30 01:37:41.754307383 +0000 UTC m=+1185.332594719" Oct 30 01:37:49.394882 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:49.394844 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f7bdcc-7676888b95-hncp6"] Oct 30 01:37:49.398262 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:49.398243 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7bdcc-7676888b95-hncp6" Oct 30 01:37:49.419322 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:49.419270 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f7bdcc-7676888b95-hncp6"] Oct 30 01:37:49.427469 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:49.427413 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f7bdcc\"" Oct 30 01:37:49.511575 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:49.511057 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f7bdcc-dockercfg-bkj47\"" Oct 30 01:37:49.615732 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:49.612291 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59112cc2-2897-43f7-b2e9-1c01c9f17687-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f7bdcc-7676888b95-hncp6\" (UID: \"59112cc2-2897-43f7-b2e9-1c01c9f17687\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7bdcc-7676888b95-hncp6" Oct 30 01:37:49.617132 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:49.617104 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4mxh\" (UniqueName: \"kubernetes.io/projected/59112cc2-2897-43f7-b2e9-1c01c9f17687-kube-api-access-g4mxh\") pod \"payload-pre-processing-e2e-ipp-a-f7bdcc-7676888b95-hncp6\" (UID: \"59112cc2-2897-43f7-b2e9-1c01c9f17687\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7bdcc-7676888b95-hncp6" Oct 30 01:37:49.726210 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:49.722837 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59112cc2-2897-43f7-b2e9-1c01c9f17687-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f7bdcc-7676888b95-hncp6\" (UID: \"59112cc2-2897-43f7-b2e9-1c01c9f17687\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7bdcc-7676888b95-hncp6" Oct 30 01:37:49.726210 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:49.722995 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4mxh\" (UniqueName: \"kubernetes.io/projected/59112cc2-2897-43f7-b2e9-1c01c9f17687-kube-api-access-g4mxh\") pod \"payload-pre-processing-e2e-ipp-a-f7bdcc-7676888b95-hncp6\" (UID: \"59112cc2-2897-43f7-b2e9-1c01c9f17687\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7bdcc-7676888b95-hncp6" Oct 30 01:37:49.764459 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:49.764316 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59112cc2-2897-43f7-b2e9-1c01c9f17687-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f7bdcc-7676888b95-hncp6\" (UID: \"59112cc2-2897-43f7-b2e9-1c01c9f17687\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7bdcc-7676888b95-hncp6" Oct 30 01:37:49.789061 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:49.785272 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4mxh\" (UniqueName: \"kubernetes.io/projected/59112cc2-2897-43f7-b2e9-1c01c9f17687-kube-api-access-g4mxh\") pod \"payload-pre-processing-e2e-ipp-a-f7bdcc-7676888b95-hncp6\" (UID: \"59112cc2-2897-43f7-b2e9-1c01c9f17687\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7bdcc-7676888b95-hncp6" Oct 30 01:37:50.010325 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:50.010240 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7bdcc-7676888b95-hncp6" Oct 30 01:37:50.360596 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:50.353619 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f7bdcc-7676888b95-hncp6"] Oct 30 01:37:50.866865 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:50.866828 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7bdcc-7676888b95-hncp6" event={"ID":"59112cc2-2897-43f7-b2e9-1c01c9f17687","Type":"ContainerStarted","Data":"f81de4590562db58c57319d57452f127a65752228e4a4f21e64e8a414a10b35b"} Oct 30 01:37:50.867370 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:50.866911 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7bdcc-7676888b95-hncp6" Oct 30 01:37:50.867370 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:50.866926 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7bdcc-7676888b95-hncp6" event={"ID":"59112cc2-2897-43f7-b2e9-1c01c9f17687","Type":"ContainerStarted","Data":"c5dbfb1c4817eca3f062b0279fa080f58d7e557c970ed8d5ac5a77e8550efe0d"} Oct 30 01:37:50.909346 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:50.909296 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7bdcc-7676888b95-hncp6" podStartSLOduration=1.909282787 podStartE2EDuration="1.909282787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:37:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:37:50.907923032 +0000 UTC m=+1194.486210368" watchObservedRunningTime="2026-10-30 01:37:50.909282787 +0000 UTC m=+1194.487570122" Oct 30 01:37:51.494106 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:37:51.494069 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:37:52.723195 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:52.723161 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-19a5f1-5996c85f9d-5j4nj" Oct 30 01:37:56.087657 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:56.087622 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-19a5f1-56c7d78bc7-4bp7l"] Oct 30 01:37:56.090773 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:56.090757 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-19a5f1-56c7d78bc7-4bp7l" Oct 30 01:37:56.109064 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:56.109039 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-19a5f1-56c7d78bc7-4bp7l"] Oct 30 01:37:56.118430 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:56.118396 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-19a5f1-serving-cert\"" Oct 30 01:37:56.193450 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:56.193400 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2eebf2d-067e-4524-a8d2-eb97208c89e4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-19a5f1-56c7d78bc7-4bp7l\" (UID: \"a2eebf2d-067e-4524-a8d2-eb97208c89e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-19a5f1-56c7d78bc7-4bp7l" Oct 30 01:37:56.193671 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:56.193516 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s525f\" (UniqueName: \"kubernetes.io/projected/a2eebf2d-067e-4524-a8d2-eb97208c89e4-kube-api-access-s525f\") pod \"maas-api-e2e-ipp-a-19a5f1-56c7d78bc7-4bp7l\" (UID: \"a2eebf2d-067e-4524-a8d2-eb97208c89e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-19a5f1-56c7d78bc7-4bp7l" Oct 30 01:37:56.193671 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:56.193650 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2eebf2d-067e-4524-a8d2-eb97208c89e4-metrics-certs\") pod \"maas-api-e2e-ipp-a-19a5f1-56c7d78bc7-4bp7l\" (UID: \"a2eebf2d-067e-4524-a8d2-eb97208c89e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-19a5f1-56c7d78bc7-4bp7l" Oct 30 01:37:56.193803 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:56.193788 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2eebf2d-067e-4524-a8d2-eb97208c89e4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-19a5f1-56c7d78bc7-4bp7l\" (UID: \"a2eebf2d-067e-4524-a8d2-eb97208c89e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-19a5f1-56c7d78bc7-4bp7l" Oct 30 01:37:56.295390 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:56.295357 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2eebf2d-067e-4524-a8d2-eb97208c89e4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-19a5f1-56c7d78bc7-4bp7l\" (UID: \"a2eebf2d-067e-4524-a8d2-eb97208c89e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-19a5f1-56c7d78bc7-4bp7l" Oct 30 01:37:56.295577 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:56.295425 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s525f\" (UniqueName: \"kubernetes.io/projected/a2eebf2d-067e-4524-a8d2-eb97208c89e4-kube-api-access-s525f\") pod \"maas-api-e2e-ipp-a-19a5f1-56c7d78bc7-4bp7l\" (UID: \"a2eebf2d-067e-4524-a8d2-eb97208c89e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-19a5f1-56c7d78bc7-4bp7l" Oct 30 01:37:56.295577 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:56.295497 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2eebf2d-067e-4524-a8d2-eb97208c89e4-metrics-certs\") pod \"maas-api-e2e-ipp-a-19a5f1-56c7d78bc7-4bp7l\" (UID: \"a2eebf2d-067e-4524-a8d2-eb97208c89e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-19a5f1-56c7d78bc7-4bp7l" Oct 30 01:37:56.295713 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:56.295651 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2eebf2d-067e-4524-a8d2-eb97208c89e4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-19a5f1-56c7d78bc7-4bp7l\" (UID: \"a2eebf2d-067e-4524-a8d2-eb97208c89e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-19a5f1-56c7d78bc7-4bp7l" Oct 30 01:37:56.296261 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:56.296240 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2eebf2d-067e-4524-a8d2-eb97208c89e4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-19a5f1-56c7d78bc7-4bp7l\" (UID: \"a2eebf2d-067e-4524-a8d2-eb97208c89e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-19a5f1-56c7d78bc7-4bp7l" Oct 30 01:37:56.297873 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:56.297848 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2eebf2d-067e-4524-a8d2-eb97208c89e4-metrics-certs\") pod \"maas-api-e2e-ipp-a-19a5f1-56c7d78bc7-4bp7l\" (UID: \"a2eebf2d-067e-4524-a8d2-eb97208c89e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-19a5f1-56c7d78bc7-4bp7l" Oct 30 01:37:56.298007 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:56.297990 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2eebf2d-067e-4524-a8d2-eb97208c89e4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-19a5f1-56c7d78bc7-4bp7l\" (UID: \"a2eebf2d-067e-4524-a8d2-eb97208c89e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-19a5f1-56c7d78bc7-4bp7l" Oct 30 01:37:56.316268 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:56.316245 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s525f\" (UniqueName: \"kubernetes.io/projected/a2eebf2d-067e-4524-a8d2-eb97208c89e4-kube-api-access-s525f\") pod \"maas-api-e2e-ipp-a-19a5f1-56c7d78bc7-4bp7l\" (UID: \"a2eebf2d-067e-4524-a8d2-eb97208c89e4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-19a5f1-56c7d78bc7-4bp7l" Oct 30 01:37:56.407880 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:56.407794 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-19a5f1-56c7d78bc7-4bp7l" Oct 30 01:37:56.680993 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:56.680912 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2vsns_d332bbb4-c699-44e8-96ba-516d8ca03e95/ovn-acl-logging/0.log" Oct 30 01:37:56.697208 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:56.697181 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2vsns_d332bbb4-c699-44e8-96ba-516d8ca03e95/ovn-acl-logging/0.log" Oct 30 01:37:56.807309 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:56.807286 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-19a5f1-56c7d78bc7-4bp7l"] Oct 30 01:37:56.809385 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:37:56.809361 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2eebf2d_067e_4524_a8d2_eb97208c89e4.slice/crio-92bb68460188fe54781c1d44bd37ba5e8deb73e03e60e6c820194c2afa1ceaa1 WatchSource:0}: Error finding container 92bb68460188fe54781c1d44bd37ba5e8deb73e03e60e6c820194c2afa1ceaa1: Status 404 returned error can't find the container with id 92bb68460188fe54781c1d44bd37ba5e8deb73e03e60e6c820194c2afa1ceaa1 Oct 30 01:37:56.810785 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:56.810764 6221 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:37:56.919704 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:56.919673 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-19a5f1-56c7d78bc7-4bp7l" event={"ID":"a2eebf2d-067e-4524-a8d2-eb97208c89e4","Type":"ContainerStarted","Data":"92bb68460188fe54781c1d44bd37ba5e8deb73e03e60e6c820194c2afa1ceaa1"} Oct 30 01:37:57.928312 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:57.928279 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-19a5f1-56c7d78bc7-4bp7l" event={"ID":"a2eebf2d-067e-4524-a8d2-eb97208c89e4","Type":"ContainerStarted","Data":"0d8a364542cd5799bc8ae15a8ef9827aad96c68962f0aacfd63ee7e71b641f76"} Oct 30 01:37:57.928907 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:57.928356 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-19a5f1-56c7d78bc7-4bp7l" Oct 30 01:37:57.977128 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:57.977078 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-19a5f1-56c7d78bc7-4bp7l" podStartSLOduration=1.6079941949999998 podStartE2EDuration="1.977064944s" totalImagesPullingTime="369.070749ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:37:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:37:57.97480947 +0000 UTC m=+1201.553096805" watchObservedRunningTime="2026-10-30 01:37:57.977064944 +0000 UTC m=+1201.555352280" Oct 30 01:37:59.695538 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:59.695488 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-666b0b-75595758bd-ls7kp"] Oct 30 01:37:59.695944 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:59.695709 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-666b0b-75595758bd-ls7kp" podUID="06a10c0f-09bf-4e42-b580-15b4fc354b59" containerName="maas-api" containerID="cri-o://a3a2888000833f9e9b42b35e974af8b0ceba2c5202ca5b738b37bef2859a757d" gracePeriod=30 Oct 30 01:37:59.942165 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:59.942134 6221 generic.go:356] "Generic (PLEG): container finished" podID="06a10c0f-09bf-4e42-b580-15b4fc354b59" containerID="a3a2888000833f9e9b42b35e974af8b0ceba2c5202ca5b738b37bef2859a757d" exitCode=0 Oct 30 01:37:59.942301 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:37:59.942216 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-666b0b-75595758bd-ls7kp" event={"ID":"06a10c0f-09bf-4e42-b580-15b4fc354b59","Type":"ContainerDied","Data":"a3a2888000833f9e9b42b35e974af8b0ceba2c5202ca5b738b37bef2859a757d"} Oct 30 01:38:00.055555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:00.049605 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-666b0b-75595758bd-ls7kp" Oct 30 01:38:00.256631 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:00.228047 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06a10c0f-09bf-4e42-b580-15b4fc354b59-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06a10c0f-09bf-4e42-b580-15b4fc354b59-maas-api-tls\") pod \"06a10c0f-09bf-4e42-b580-15b4fc354b59\" (UID: \"06a10c0f-09bf-4e42-b580-15b4fc354b59\") " Oct 30 01:38:00.256631 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:00.228312 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06a10c0f-09bf-4e42-b580-15b4fc354b59-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06a10c0f-09bf-4e42-b580-15b4fc354b59-openshift-service-ca\") pod \"06a10c0f-09bf-4e42-b580-15b4fc354b59\" (UID: \"06a10c0f-09bf-4e42-b580-15b4fc354b59\") " Oct 30 01:38:00.256631 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:00.228351 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06a10c0f-09bf-4e42-b580-15b4fc354b59-metrics-certs\" (UniqueName: \"kubernetes.io/secret/06a10c0f-09bf-4e42-b580-15b4fc354b59-metrics-certs\") pod \"06a10c0f-09bf-4e42-b580-15b4fc354b59\" (UID: \"06a10c0f-09bf-4e42-b580-15b4fc354b59\") " Oct 30 01:38:00.256631 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:00.228518 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06a10c0f-09bf-4e42-b580-15b4fc354b59-kube-api-access-wcwht\" (UniqueName: \"kubernetes.io/projected/06a10c0f-09bf-4e42-b580-15b4fc354b59-kube-api-access-wcwht\") pod \"06a10c0f-09bf-4e42-b580-15b4fc354b59\" (UID: \"06a10c0f-09bf-4e42-b580-15b4fc354b59\") " Oct 30 01:38:00.256631 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:00.253136 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06a10c0f-09bf-4e42-b580-15b4fc354b59-openshift-service-ca" pod "06a10c0f-09bf-4e42-b580-15b4fc354b59" (UID: "06a10c0f-09bf-4e42-b580-15b4fc354b59"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:38:00.279553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:00.263911 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06a10c0f-09bf-4e42-b580-15b4fc354b59-kube-api-access-wcwht" pod "06a10c0f-09bf-4e42-b580-15b4fc354b59" (UID: "06a10c0f-09bf-4e42-b580-15b4fc354b59"). InnerVolumeSpecName "kube-api-access-wcwht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:00.300570 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:00.289886 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06a10c0f-09bf-4e42-b580-15b4fc354b59-maas-api-tls" pod "06a10c0f-09bf-4e42-b580-15b4fc354b59" (UID: "06a10c0f-09bf-4e42-b580-15b4fc354b59"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:38:00.300570 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:00.290937 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06a10c0f-09bf-4e42-b580-15b4fc354b59-metrics-certs" pod "06a10c0f-09bf-4e42-b580-15b4fc354b59" (UID: "06a10c0f-09bf-4e42-b580-15b4fc354b59"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:38:00.498081 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:00.463669 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcwht\" (UniqueName: \"kubernetes.io/projected/06a10c0f-09bf-4e42-b580-15b4fc354b59-kube-api-access-wcwht\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:38:00.498081 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:00.463708 6221 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/06a10c0f-09bf-4e42-b580-15b4fc354b59-maas-api-tls\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:38:00.498081 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:00.463723 6221 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/06a10c0f-09bf-4e42-b580-15b4fc354b59-openshift-service-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:38:00.498081 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:00.463737 6221 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/06a10c0f-09bf-4e42-b580-15b4fc354b59-metrics-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:38:00.990751 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:00.975560 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-666b0b-75595758bd-ls7kp" event={"ID":"06a10c0f-09bf-4e42-b580-15b4fc354b59","Type":"ContainerDied","Data":"d24264a8761effdae4413b62ffe8d4313a4e7dbecb5a65e548d92524728fa1ad"} Oct 30 01:38:00.990751 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:00.975661 6221 scope.go:122] "RemoveContainer" containerID="a3a2888000833f9e9b42b35e974af8b0ceba2c5202ca5b738b37bef2859a757d" Oct 30 01:38:00.990751 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:00.975883 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-666b0b-75595758bd-ls7kp" Oct 30 01:38:01.044647 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:01.044613 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-666b0b-75595758bd-ls7kp"] Oct 30 01:38:01.048543 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:01.048507 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-666b0b-75595758bd-ls7kp"] Oct 30 01:38:01.430993 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:01.430955 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f09934-98df65955-8cwfc"] Oct 30 01:38:01.876790 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:01.876758 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7bdcc-7676888b95-hncp6" Oct 30 01:38:02.547773 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:02.547735 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06a10c0f-09bf-4e42-b580-15b4fc354b59" path="/var/lib/kubelet/pods/06a10c0f-09bf-4e42-b580-15b4fc354b59/volumes" Oct 30 01:38:03.939705 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:03.939674 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-19a5f1-56c7d78bc7-4bp7l" Oct 30 01:38:06.431616 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:06.431573 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f09934-98df65955-8cwfc" podUID="8ece0817-69ac-4fa5-a10b-67b88b368b51" containerName="payload-pre-processing" containerID="cri-o://be4a2271e9173ee5d0c069d7ca596b3adde0b915fc1c8eedb7057d2582ab2a39" gracePeriod=25 Oct 30 01:38:06.815509 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:06.815477 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f09934-98df65955-8cwfc" Oct 30 01:38:06.982256 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:06.982164 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ece0817-69ac-4fa5-a10b-67b88b368b51-kube-api-access-xs8r7\" (UniqueName: \"kubernetes.io/projected/8ece0817-69ac-4fa5-a10b-67b88b368b51-kube-api-access-xs8r7\") pod \"8ece0817-69ac-4fa5-a10b-67b88b368b51\" (UID: \"8ece0817-69ac-4fa5-a10b-67b88b368b51\") " Oct 30 01:38:06.982437 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:06.982262 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ece0817-69ac-4fa5-a10b-67b88b368b51-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ece0817-69ac-4fa5-a10b-67b88b368b51-plugins-config-volume\") pod \"8ece0817-69ac-4fa5-a10b-67b88b368b51\" (UID: \"8ece0817-69ac-4fa5-a10b-67b88b368b51\") " Oct 30 01:38:06.982696 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:06.982668 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ece0817-69ac-4fa5-a10b-67b88b368b51-plugins-config-volume" pod "8ece0817-69ac-4fa5-a10b-67b88b368b51" (UID: "8ece0817-69ac-4fa5-a10b-67b88b368b51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:38:06.982996 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:06.982978 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ece0817-69ac-4fa5-a10b-67b88b368b51-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:38:06.984265 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:06.984244 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ece0817-69ac-4fa5-a10b-67b88b368b51-kube-api-access-xs8r7" pod "8ece0817-69ac-4fa5-a10b-67b88b368b51" (UID: "8ece0817-69ac-4fa5-a10b-67b88b368b51"). InnerVolumeSpecName "kube-api-access-xs8r7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:07.077370 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:07.077335 6221 generic.go:356] "Generic (PLEG): container finished" podID="8ece0817-69ac-4fa5-a10b-67b88b368b51" containerID="be4a2271e9173ee5d0c069d7ca596b3adde0b915fc1c8eedb7057d2582ab2a39" exitCode=0 Oct 30 01:38:07.077370 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:07.077381 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f09934-98df65955-8cwfc" Oct 30 01:38:07.077618 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:07.077449 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f09934-98df65955-8cwfc" event={"ID":"8ece0817-69ac-4fa5-a10b-67b88b368b51","Type":"ContainerDied","Data":"be4a2271e9173ee5d0c069d7ca596b3adde0b915fc1c8eedb7057d2582ab2a39"} Oct 30 01:38:07.077618 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:07.077504 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f09934-98df65955-8cwfc" event={"ID":"8ece0817-69ac-4fa5-a10b-67b88b368b51","Type":"ContainerDied","Data":"3ee6b88a33d9fc3f7391b7aeec41d51f625e27f0f27e2459f1a1d88c7b91ae2f"} Oct 30 01:38:07.077618 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:07.077526 6221 scope.go:122] "RemoveContainer" containerID="be4a2271e9173ee5d0c069d7ca596b3adde0b915fc1c8eedb7057d2582ab2a39" Oct 30 01:38:07.084027 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:07.084001 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs8r7\" (UniqueName: \"kubernetes.io/projected/8ece0817-69ac-4fa5-a10b-67b88b368b51-kube-api-access-xs8r7\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:38:07.089281 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:07.089264 6221 scope.go:122] "RemoveContainer" containerID="be4a2271e9173ee5d0c069d7ca596b3adde0b915fc1c8eedb7057d2582ab2a39" Oct 30 01:38:07.089601 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:38:07.089575 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be4a2271e9173ee5d0c069d7ca596b3adde0b915fc1c8eedb7057d2582ab2a39\": container with ID starting with be4a2271e9173ee5d0c069d7ca596b3adde0b915fc1c8eedb7057d2582ab2a39 not found: ID does not exist" containerID="be4a2271e9173ee5d0c069d7ca596b3adde0b915fc1c8eedb7057d2582ab2a39" Oct 30 01:38:07.089688 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:07.089613 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be4a2271e9173ee5d0c069d7ca596b3adde0b915fc1c8eedb7057d2582ab2a39"} err="failed to get container status \"be4a2271e9173ee5d0c069d7ca596b3adde0b915fc1c8eedb7057d2582ab2a39\": rpc error: code = NotFound desc = could not find container \"be4a2271e9173ee5d0c069d7ca596b3adde0b915fc1c8eedb7057d2582ab2a39\": container with ID starting with be4a2271e9173ee5d0c069d7ca596b3adde0b915fc1c8eedb7057d2582ab2a39 not found: ID does not exist" Oct 30 01:38:07.132600 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:07.132557 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f09934-98df65955-8cwfc"] Oct 30 01:38:07.136157 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:07.136137 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f09934-98df65955-8cwfc"] Oct 30 01:38:07.759951 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:07.759920 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg"] Oct 30 01:38:07.760446 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:07.760188 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" podUID="fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab" containerName="istio-proxy" containerID="cri-o://28b4e7545449943383c9f91713bca136811f3b234c81bd81a7c4c27d4b882c4a" gracePeriod=30 Oct 30 01:38:08.277385 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:08.277341 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7bdcc-57cf44cdf5-x2vkk"] Oct 30 01:38:08.278071 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:08.278050 6221 cpu_manager.go:414] "containerMap: removing container" podUID="06a10c0f-09bf-4e42-b580-15b4fc354b59" containerName="maas-api" Oct 30 01:38:08.278174 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:08.278072 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06a10c0f-09bf-4e42-b580-15b4fc354b59" containerName="maas-api" Oct 30 01:38:08.278174 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:08.278109 6221 cpu_manager.go:414] "containerMap: removing container" podUID="8ece0817-69ac-4fa5-a10b-67b88b368b51" containerName="payload-pre-processing" Oct 30 01:38:08.278174 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:08.278117 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ece0817-69ac-4fa5-a10b-67b88b368b51" containerName="payload-pre-processing" Oct 30 01:38:08.278343 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:08.278282 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="06a10c0f-09bf-4e42-b580-15b4fc354b59" containerName="maas-api" Oct 30 01:38:08.278343 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:08.278301 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ece0817-69ac-4fa5-a10b-67b88b368b51" containerName="payload-pre-processing" Oct 30 01:38:08.283392 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:08.283369 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7bdcc-57cf44cdf5-x2vkk" Oct 30 01:38:08.299140 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:08.299120 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-f7bdcc-serving-cert\"" Oct 30 01:38:08.302516 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:08.302491 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7bdcc-57cf44cdf5-x2vkk"] Oct 30 01:38:08.415111 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:08.415069 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfnrv\" (UniqueName: \"kubernetes.io/projected/41ea8723-6c14-44b1-8926-ae762a267b6e-kube-api-access-gfnrv\") pod \"maas-api-e2e-ipp-a-f7bdcc-57cf44cdf5-x2vkk\" (UID: \"41ea8723-6c14-44b1-8926-ae762a267b6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7bdcc-57cf44cdf5-x2vkk" Oct 30 01:38:08.415264 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:08.415147 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41ea8723-6c14-44b1-8926-ae762a267b6e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f7bdcc-57cf44cdf5-x2vkk\" (UID: \"41ea8723-6c14-44b1-8926-ae762a267b6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7bdcc-57cf44cdf5-x2vkk" Oct 30 01:38:08.415264 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:08.415177 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41ea8723-6c14-44b1-8926-ae762a267b6e-metrics-certs\") pod \"maas-api-e2e-ipp-a-f7bdcc-57cf44cdf5-x2vkk\" (UID: \"41ea8723-6c14-44b1-8926-ae762a267b6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7bdcc-57cf44cdf5-x2vkk" Oct 30 01:38:08.415361 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:08.415258 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41ea8723-6c14-44b1-8926-ae762a267b6e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f7bdcc-57cf44cdf5-x2vkk\" (UID: \"41ea8723-6c14-44b1-8926-ae762a267b6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7bdcc-57cf44cdf5-x2vkk" Oct 30 01:38:08.517116 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:08.517082 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfnrv\" (UniqueName: \"kubernetes.io/projected/41ea8723-6c14-44b1-8926-ae762a267b6e-kube-api-access-gfnrv\") pod \"maas-api-e2e-ipp-a-f7bdcc-57cf44cdf5-x2vkk\" (UID: \"41ea8723-6c14-44b1-8926-ae762a267b6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7bdcc-57cf44cdf5-x2vkk" Oct 30 01:38:08.517274 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:08.517147 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41ea8723-6c14-44b1-8926-ae762a267b6e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f7bdcc-57cf44cdf5-x2vkk\" (UID: \"41ea8723-6c14-44b1-8926-ae762a267b6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7bdcc-57cf44cdf5-x2vkk" Oct 30 01:38:08.517340 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:08.517309 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41ea8723-6c14-44b1-8926-ae762a267b6e-metrics-certs\") pod \"maas-api-e2e-ipp-a-f7bdcc-57cf44cdf5-x2vkk\" (UID: \"41ea8723-6c14-44b1-8926-ae762a267b6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7bdcc-57cf44cdf5-x2vkk" Oct 30 01:38:08.517387 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:08.517349 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41ea8723-6c14-44b1-8926-ae762a267b6e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f7bdcc-57cf44cdf5-x2vkk\" (UID: \"41ea8723-6c14-44b1-8926-ae762a267b6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7bdcc-57cf44cdf5-x2vkk" Oct 30 01:38:08.518068 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:08.518043 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41ea8723-6c14-44b1-8926-ae762a267b6e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f7bdcc-57cf44cdf5-x2vkk\" (UID: \"41ea8723-6c14-44b1-8926-ae762a267b6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7bdcc-57cf44cdf5-x2vkk" Oct 30 01:38:08.519870 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:08.519842 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41ea8723-6c14-44b1-8926-ae762a267b6e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f7bdcc-57cf44cdf5-x2vkk\" (UID: \"41ea8723-6c14-44b1-8926-ae762a267b6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7bdcc-57cf44cdf5-x2vkk" Oct 30 01:38:08.519967 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:08.519843 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41ea8723-6c14-44b1-8926-ae762a267b6e-metrics-certs\") pod \"maas-api-e2e-ipp-a-f7bdcc-57cf44cdf5-x2vkk\" (UID: \"41ea8723-6c14-44b1-8926-ae762a267b6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7bdcc-57cf44cdf5-x2vkk" Oct 30 01:38:08.615465 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:08.615423 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ece0817-69ac-4fa5-a10b-67b88b368b51" path="/var/lib/kubelet/pods/8ece0817-69ac-4fa5-a10b-67b88b368b51/volumes" Oct 30 01:38:08.621248 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:08.621214 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfnrv\" (UniqueName: \"kubernetes.io/projected/41ea8723-6c14-44b1-8926-ae762a267b6e-kube-api-access-gfnrv\") pod \"maas-api-e2e-ipp-a-f7bdcc-57cf44cdf5-x2vkk\" (UID: \"41ea8723-6c14-44b1-8926-ae762a267b6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7bdcc-57cf44cdf5-x2vkk" Oct 30 01:38:08.662622 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:08.661522 6221 patch_prober.go:29] interesting pod/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:38:08.662622 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:08.661616 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" podUID="fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:38:08.897567 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:08.897471 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7bdcc-57cf44cdf5-x2vkk" Oct 30 01:38:09.213204 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:09.210476 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7bdcc-57cf44cdf5-x2vkk"] Oct 30 01:38:09.213438 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:38:09.213399 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41ea8723_6c14_44b1_8926_ae762a267b6e.slice/crio-ed03c563738fbdc838c503c908a9ffeddcf7887d7d7b307e58fe19176f0d2bff WatchSource:0}: Error finding container ed03c563738fbdc838c503c908a9ffeddcf7887d7d7b307e58fe19176f0d2bff: Status 404 returned error can't find the container with id ed03c563738fbdc838c503c908a9ffeddcf7887d7d7b307e58fe19176f0d2bff Oct 30 01:38:09.281378 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:09.281346 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:38:10.124799 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:10.124705 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7bdcc-57cf44cdf5-x2vkk" event={"ID":"41ea8723-6c14-44b1-8926-ae762a267b6e","Type":"ContainerStarted","Data":"6ff3151435cc343516d8dfa50aad4d108ff7809fa875929f179fc4f5515bdfea"} Oct 30 01:38:10.124799 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:10.124757 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7bdcc-57cf44cdf5-x2vkk" event={"ID":"41ea8723-6c14-44b1-8926-ae762a267b6e","Type":"ContainerStarted","Data":"ed03c563738fbdc838c503c908a9ffeddcf7887d7d7b307e58fe19176f0d2bff"} Oct 30 01:38:10.125318 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:10.124827 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7bdcc-57cf44cdf5-x2vkk" Oct 30 01:38:10.167375 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:10.167318 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7bdcc-57cf44cdf5-x2vkk" podStartSLOduration=1.570085224 podStartE2EDuration="2.167301268s" totalImagesPullingTime="597.216044ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:38:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:38:10.165379484 +0000 UTC m=+1213.743666820" watchObservedRunningTime="2026-10-30 01:38:10.167301268 +0000 UTC m=+1213.745588642" Oct 30 01:38:12.406559 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:12.401622 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" podUID="9df52a01-0c8d-44ba-9fd7-9cb2fa41947a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:38:12.750327 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:12.682614 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" podUID="354ec282-b972-4fca-b4e3-1e33ef461f2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 01:38:12.750327 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:12.736553 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" podUID="018c3fe2-3e67-45ed-a3d7-d70f3fe5b956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:38:12.750327 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:12.737738 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" podUID="3b3c0e08-1371-4184-816a-d8b501526190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:38:12.776661 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:12.775861 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" podUID="ea9583aa-8259-4c9b-932e-2d6559008d86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:38:12.797028 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:12.796973 6221 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" podUID="3b3c0e08-1371-4184-816a-d8b501526190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:38:13.032054 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:13.031638 6221 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" podUID="354ec282-b972-4fca-b4e3-1e33ef461f2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:38:13.241601 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:38:13.239590 6221 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfefc3c51_c7f6_4f1c_85dd_2c23ab8a99ab.slice/crio-conmon-28b4e7545449943383c9f91713bca136811f3b234c81bd81a7c4c27d4b882c4a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfefc3c51_c7f6_4f1c_85dd_2c23ab8a99ab.slice/crio-28b4e7545449943383c9f91713bca136811f3b234c81bd81a7c4c27d4b882c4a.scope\": RecentStats: unable to find data in memory cache]" Oct 30 01:38:13.269315 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:38:13.269186 6221 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfefc3c51_c7f6_4f1c_85dd_2c23ab8a99ab.slice/crio-conmon-28b4e7545449943383c9f91713bca136811f3b234c81bd81a7c4c27d4b882c4a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfefc3c51_c7f6_4f1c_85dd_2c23ab8a99ab.slice/crio-28b4e7545449943383c9f91713bca136811f3b234c81bd81a7c4c27d4b882c4a.scope\": RecentStats: unable to find data in memory cache]" Oct 30 01:38:13.420558 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:13.396221 6221 generic.go:356] "Generic (PLEG): container finished" podID="fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab" containerID="28b4e7545449943383c9f91713bca136811f3b234c81bd81a7c4c27d4b882c4a" exitCode=0 Oct 30 01:38:13.420558 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:13.396287 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" event={"ID":"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab","Type":"ContainerDied","Data":"28b4e7545449943383c9f91713bca136811f3b234c81bd81a7c4c27d4b882c4a"} Oct 30 01:38:13.420558 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:13.396351 6221 scope.go:122] "RemoveContainer" containerID="3bd4319c2a6fd48570dcd57f73bab2ce1dd6117f44f272b20e2415cdf5f1b2fd" Oct 30 01:38:13.420558 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:13.402144 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" podUID="9df52a01-0c8d-44ba-9fd7-9cb2fa41947a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:38:13.420558 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:13.402909 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" Oct 30 01:38:13.715651 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:13.703686 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk" podUID="f4ae494e-81ec-4039-ae07-7ed1bf345f0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:38:13.741828 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:13.706343 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" podUID="354ec282-b972-4fca-b4e3-1e33ef461f2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 01:38:13.760614 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:13.743215 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" Oct 30 01:38:13.760614 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:13.741356 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" podUID="3b3c0e08-1371-4184-816a-d8b501526190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:38:13.760614 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:13.741388 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" podUID="018c3fe2-3e67-45ed-a3d7-d70f3fe5b956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 01:38:13.760614 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:13.746334 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" Oct 30 01:38:13.760614 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:13.746383 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" Oct 30 01:38:13.760614 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:13.749429 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" Oct 30 01:38:13.787554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:13.777928 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" podUID="ea9583aa-8259-4c9b-932e-2d6559008d86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:38:13.787554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:13.779122 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" Oct 30 01:38:14.116560 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:14.111248 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" Oct 30 01:38:14.199252 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:14.173931 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" Oct 30 01:38:14.276953 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:14.273502 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" Oct 30 01:38:14.291858 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:14.291829 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" Oct 30 01:38:14.379962 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:14.379879 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" Oct 30 01:38:14.404977 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:14.404950 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" Oct 30 01:38:14.405148 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:14.405006 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg" event={"ID":"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab","Type":"ContainerDied","Data":"3ebc97722c49cb06e3dbc45b5a5a2553ea08cf93be2dc8da68f2290d0db5a62a"} Oct 30 01:38:14.405148 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:14.405068 6221 scope.go:122] "RemoveContainer" containerID="28b4e7545449943383c9f91713bca136811f3b234c81bd81a7c4c27d4b882c4a" Oct 30 01:38:14.406309 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:14.406285 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-credential-socket\") pod \"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab\" (UID: \"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab\") " Oct 30 01:38:14.406397 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:14.406342 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-workload-certs\") pod \"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab\" (UID: \"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab\") " Oct 30 01:38:14.406454 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:14.406415 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-kube-api-access-tc9xd\" (UniqueName: \"kubernetes.io/projected/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-kube-api-access-tc9xd\") pod \"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab\" (UID: \"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab\") " Oct 30 01:38:14.406510 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:14.406470 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-workload-socket\") pod \"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab\" (UID: \"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab\") " Oct 30 01:38:14.406586 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:14.406559 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-istio-token\" (UniqueName: \"kubernetes.io/projected/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-istio-token\") pod \"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab\" (UID: \"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab\") " Oct 30 01:38:14.406639 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:14.406620 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-credential-socket" pod "fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab" (UID: "fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:14.419674 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:14.419637 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-istio-envoy\") pod \"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab\" (UID: \"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab\") " Oct 30 01:38:14.425552 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:14.419962 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-istio-data\") pod \"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab\" (UID: \"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab\") " Oct 30 01:38:14.451761 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:14.424408 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-workload-certs" pod "fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab" (UID: "fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:14.451761 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:14.430610 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-workload-socket" pod "fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab" (UID: "fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:14.451761 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:14.440623 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-istio-data" pod "fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab" (UID: "fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:14.451761 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:14.443509 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-istio-envoy" pod "fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab" (UID: "fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:14.451761 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:14.443887 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-istio-token" pod "fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab" (UID: "fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:14.451761 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:14.446738 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-kube-api-access-tc9xd" pod "fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab" (UID: "fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab"). InnerVolumeSpecName "kube-api-access-tc9xd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:14.451761 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:14.450755 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-istio-podinfo\") pod \"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab\" (UID: \"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab\") " Oct 30 01:38:14.451761 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:14.450871 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-istiod-ca-cert\") pod \"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab\" (UID: \"fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab\") " Oct 30 01:38:14.451761 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:14.451446 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-istiod-ca-cert" pod "fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab" (UID: "fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:38:14.452228 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:14.451777 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:38:14.452228 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:14.451802 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:38:14.452228 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:14.451819 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc9xd\" (UniqueName: \"kubernetes.io/projected/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-kube-api-access-tc9xd\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:38:14.452228 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:14.451833 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:38:14.452228 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:14.451846 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:38:14.452228 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:14.451859 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:38:14.452228 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:14.451872 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:38:14.452228 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:14.451884 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:38:14.456724 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:14.456678 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-istio-podinfo" pod "fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab" (UID: "fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:38:14.553613 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:14.553569 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:38:14.781593 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:14.781527 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg"] Oct 30 01:38:14.797433 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:14.797388 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-666b0b-openshift-default-57977c895f-fmsmg"] Oct 30 01:38:15.337017 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:15.336965 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6b3dbd-6f8f7bfddb-8t572"] Oct 30 01:38:15.337699 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:15.337678 6221 cpu_manager.go:414] "containerMap: removing container" podUID="fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab" containerName="istio-proxy" Oct 30 01:38:15.337699 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:15.337697 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab" containerName="istio-proxy" Oct 30 01:38:15.337894 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:15.337839 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab" containerName="istio-proxy" Oct 30 01:38:15.337894 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:15.337853 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab" containerName="istio-proxy" Oct 30 01:38:15.341238 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:15.341222 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3dbd-6f8f7bfddb-8t572" Oct 30 01:38:15.354723 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:15.354698 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6b3dbd-6f8f7bfddb-8t572"] Oct 30 01:38:15.358112 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:15.358088 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-6b3dbd\"" Oct 30 01:38:15.437360 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:15.437321 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3dbd-775fc65d4c-bjhqz"] Oct 30 01:38:15.442554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:15.438383 6221 cpu_manager.go:414] "containerMap: removing container" podUID="fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab" containerName="istio-proxy" Oct 30 01:38:15.442554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:15.438407 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab" containerName="istio-proxy" Oct 30 01:38:15.443402 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:15.443383 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3dbd-775fc65d4c-bjhqz" Oct 30 01:38:15.448656 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:15.448631 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3dbd-775fc65d4c-bjhqz"] Oct 30 01:38:15.460555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:15.460494 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-6b3dbd-dockercfg-nxf5s\"" Oct 30 01:38:15.472354 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:15.466583 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px8w6\" (UniqueName: \"kubernetes.io/projected/69f4a147-db8e-4b2c-a3a1-c7259e37ceba-kube-api-access-px8w6\") pod \"payload-processing-e2e-ipp-a-6b3dbd-6f8f7bfddb-8t572\" (UID: \"69f4a147-db8e-4b2c-a3a1-c7259e37ceba\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3dbd-6f8f7bfddb-8t572" Oct 30 01:38:15.472354 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:15.466719 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69f4a147-db8e-4b2c-a3a1-c7259e37ceba-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6b3dbd-6f8f7bfddb-8t572\" (UID: \"69f4a147-db8e-4b2c-a3a1-c7259e37ceba\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3dbd-6f8f7bfddb-8t572" Oct 30 01:38:15.618616 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:15.579519 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8ac61bc-e5e8-4615-9767-9991ba366b7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6b3dbd-775fc65d4c-bjhqz\" (UID: \"a8ac61bc-e5e8-4615-9767-9991ba366b7a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3dbd-775fc65d4c-bjhqz" Oct 30 01:38:15.618616 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:15.579644 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px8w6\" (UniqueName: \"kubernetes.io/projected/69f4a147-db8e-4b2c-a3a1-c7259e37ceba-kube-api-access-px8w6\") pod \"payload-processing-e2e-ipp-a-6b3dbd-6f8f7bfddb-8t572\" (UID: \"69f4a147-db8e-4b2c-a3a1-c7259e37ceba\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3dbd-6f8f7bfddb-8t572" Oct 30 01:38:15.618616 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:15.579767 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69f4a147-db8e-4b2c-a3a1-c7259e37ceba-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6b3dbd-6f8f7bfddb-8t572\" (UID: \"69f4a147-db8e-4b2c-a3a1-c7259e37ceba\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3dbd-6f8f7bfddb-8t572" Oct 30 01:38:15.618616 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:15.579800 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz74m\" (UniqueName: \"kubernetes.io/projected/a8ac61bc-e5e8-4615-9767-9991ba366b7a-kube-api-access-qz74m\") pod \"payload-pre-processing-e2e-ipp-a-6b3dbd-775fc65d4c-bjhqz\" (UID: \"a8ac61bc-e5e8-4615-9767-9991ba366b7a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3dbd-775fc65d4c-bjhqz" Oct 30 01:38:15.618616 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:15.581211 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69f4a147-db8e-4b2c-a3a1-c7259e37ceba-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-6b3dbd-6f8f7bfddb-8t572\" (UID: \"69f4a147-db8e-4b2c-a3a1-c7259e37ceba\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3dbd-6f8f7bfddb-8t572" Oct 30 01:38:15.695554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:15.695442 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz74m\" (UniqueName: \"kubernetes.io/projected/a8ac61bc-e5e8-4615-9767-9991ba366b7a-kube-api-access-qz74m\") pod \"payload-pre-processing-e2e-ipp-a-6b3dbd-775fc65d4c-bjhqz\" (UID: \"a8ac61bc-e5e8-4615-9767-9991ba366b7a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3dbd-775fc65d4c-bjhqz" Oct 30 01:38:15.695759 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:15.695718 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8ac61bc-e5e8-4615-9767-9991ba366b7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6b3dbd-775fc65d4c-bjhqz\" (UID: \"a8ac61bc-e5e8-4615-9767-9991ba366b7a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3dbd-775fc65d4c-bjhqz" Oct 30 01:38:15.718756 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:15.718720 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8ac61bc-e5e8-4615-9767-9991ba366b7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-6b3dbd-775fc65d4c-bjhqz\" (UID: \"a8ac61bc-e5e8-4615-9767-9991ba366b7a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3dbd-775fc65d4c-bjhqz" Oct 30 01:38:15.776554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:15.774684 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px8w6\" (UniqueName: \"kubernetes.io/projected/69f4a147-db8e-4b2c-a3a1-c7259e37ceba-kube-api-access-px8w6\") pod \"payload-processing-e2e-ipp-a-6b3dbd-6f8f7bfddb-8t572\" (UID: \"69f4a147-db8e-4b2c-a3a1-c7259e37ceba\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3dbd-6f8f7bfddb-8t572" Oct 30 01:38:15.799588 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:15.781971 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz74m\" (UniqueName: \"kubernetes.io/projected/a8ac61bc-e5e8-4615-9767-9991ba366b7a-kube-api-access-qz74m\") pod \"payload-pre-processing-e2e-ipp-a-6b3dbd-775fc65d4c-bjhqz\" (UID: \"a8ac61bc-e5e8-4615-9767-9991ba366b7a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3dbd-775fc65d4c-bjhqz" Oct 30 01:38:15.882914 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:15.882826 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3dbd-775fc65d4c-bjhqz" Oct 30 01:38:15.964839 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:15.964800 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3dbd-6f8f7bfddb-8t572" Oct 30 01:38:16.244586 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:16.241714 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7bdcc-57cf44cdf5-x2vkk" Oct 30 01:38:16.559510 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:16.559464 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab" path="/var/lib/kubelet/pods/fefc3c51-c7f6-4f1c-85dd-2c23ab8a99ab/volumes" Oct 30 01:38:16.681869 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:16.680898 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6b3dbd-6f8f7bfddb-8t572"] Oct 30 01:38:16.779313 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:16.779291 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3dbd-775fc65d4c-bjhqz"] Oct 30 01:38:16.782673 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:38:16.782648 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8ac61bc_e5e8_4615_9767_9991ba366b7a.slice/crio-13dec5d9d8fff04490e4f04e399c02c6ab48500fcc473ce696cd92429b874ce0 WatchSource:0}: Error finding container 13dec5d9d8fff04490e4f04e399c02c6ab48500fcc473ce696cd92429b874ce0: Status 404 returned error can't find the container with id 13dec5d9d8fff04490e4f04e399c02c6ab48500fcc473ce696cd92429b874ce0 Oct 30 01:38:17.490671 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:17.490409 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3dbd-6f8f7bfddb-8t572" event={"ID":"69f4a147-db8e-4b2c-a3a1-c7259e37ceba","Type":"ContainerStarted","Data":"80cde8c6a61690e8173225a459389407d57a97d5dc446cc7c1c5b7b658572c8d"} Oct 30 01:38:17.490671 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:17.490481 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3dbd-6f8f7bfddb-8t572" event={"ID":"69f4a147-db8e-4b2c-a3a1-c7259e37ceba","Type":"ContainerStarted","Data":"9c097cf76bf46d5e6da3c1098cb038309a4c56c142a244a210f4328b45a7a0bc"} Oct 30 01:38:17.490671 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:17.490669 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3dbd-6f8f7bfddb-8t572" Oct 30 01:38:17.491997 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:17.491970 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3dbd-775fc65d4c-bjhqz" event={"ID":"a8ac61bc-e5e8-4615-9767-9991ba366b7a","Type":"ContainerStarted","Data":"fcd9f04ab19f8d3f06f52bb166ff5ee26a7f1250bfa0072ca5c70a3fc6a2231c"} Oct 30 01:38:17.492116 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:17.492015 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3dbd-775fc65d4c-bjhqz" event={"ID":"a8ac61bc-e5e8-4615-9767-9991ba366b7a","Type":"ContainerStarted","Data":"13dec5d9d8fff04490e4f04e399c02c6ab48500fcc473ce696cd92429b874ce0"} Oct 30 01:38:17.492116 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:17.492049 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3dbd-775fc65d4c-bjhqz" Oct 30 01:38:17.532764 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:17.532700 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3dbd-6f8f7bfddb-8t572" podStartSLOduration=2.532685356 podStartE2EDuration="2.532685356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:38:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:38:17.531960111 +0000 UTC m=+1221.110247447" watchObservedRunningTime="2026-10-30 01:38:17.532685356 +0000 UTC m=+1221.110972691" Oct 30 01:38:17.574250 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:17.574198 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3dbd-775fc65d4c-bjhqz" podStartSLOduration=2.574184449 podStartE2EDuration="2.574184449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:38:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:38:17.572408099 +0000 UTC m=+1221.150695437" watchObservedRunningTime="2026-10-30 01:38:17.574184449 +0000 UTC m=+1221.152471790" Oct 30 01:38:21.607323 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:38:21.602033 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:38:21.870252 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:21.868992 6221 patch_prober.go:29] interesting pod/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": read tcp 10.134.0.2:41226->10.134.0.97:15021: read: connection reset by peer" start-of-body= Oct 30 01:38:21.870252 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:21.869075 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" podUID="43a3609a-8ccb-4149-82ec-98834c8f95a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": read tcp 10.134.0.2:41226->10.134.0.97:15021: read: connection reset by peer" Oct 30 01:38:22.593483 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:22.593449 6221 generic.go:356] "Generic (PLEG): container finished" podID="43a3609a-8ccb-4149-82ec-98834c8f95a3" containerID="3fd81a3eef3c036d1ffa2979aeba8042424968be064164f5de527228f05d70b2" exitCode=0 Oct 30 01:38:22.593681 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:22.593521 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" event={"ID":"43a3609a-8ccb-4149-82ec-98834c8f95a3","Type":"ContainerDied","Data":"3fd81a3eef3c036d1ffa2979aeba8042424968be064164f5de527228f05d70b2"} Oct 30 01:38:22.593953 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:22.593940 6221 scope.go:122] "RemoveContainer" containerID="3fd81a3eef3c036d1ffa2979aeba8042424968be064164f5de527228f05d70b2" Oct 30 01:38:22.595499 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:22.595440 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:38:22.595598 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:22.595554 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:38:22.595598 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:22.595588 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:38:23.601862 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:23.601824 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" event={"ID":"43a3609a-8ccb-4149-82ec-98834c8f95a3","Type":"ContainerStarted","Data":"222cd8b4e05700e20e788a1057953a84cdf53808bfa380db63a30d9445586484"} Oct 30 01:38:23.866603 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:23.861843 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-19a5f1-5996c85f9d-5j4nj"] Oct 30 01:38:24.589682 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:24.589648 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" Oct 30 01:38:24.590103 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:24.589889 6221 patch_prober.go:29] interesting pod/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 01:38:24.590103 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:24.589944 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" podUID="43a3609a-8ccb-4149-82ec-98834c8f95a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 01:38:25.589908 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:25.589871 6221 patch_prober.go:29] interesting pod/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 01:38:25.590402 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:25.589932 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" podUID="43a3609a-8ccb-4149-82ec-98834c8f95a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 01:38:26.589778 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:26.589744 6221 patch_prober.go:29] interesting pod/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 01:38:26.589965 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:26.589815 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" podUID="43a3609a-8ccb-4149-82ec-98834c8f95a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 01:38:26.735250 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:26.735214 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65"] Oct 30 01:38:26.735482 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:26.735446 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65" podUID="0dcc7bfb-9764-4e3e-9126-e45fa253a25e" containerName="istio-proxy" containerID="cri-o://a2853aa6af3dc7068a99b2449661db1691446efcd8074b1e44da68d4b633fa15" gracePeriod=30 Oct 30 01:38:27.597554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:27.593941 6221 patch_prober.go:29] interesting pod/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 01:38:27.597554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:27.594006 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" podUID="43a3609a-8ccb-4149-82ec-98834c8f95a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 01:38:28.501911 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:28.501881 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3dbd-6f8f7bfddb-8t572" Oct 30 01:38:28.502079 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:28.501944 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3dbd-775fc65d4c-bjhqz" Oct 30 01:38:28.589184 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:28.589149 6221 patch_prober.go:29] interesting pod/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 01:38:28.589335 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:28.589228 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" podUID="43a3609a-8ccb-4149-82ec-98834c8f95a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 01:38:28.862963 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:28.862908 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-19a5f1-5996c85f9d-5j4nj" podUID="15d60827-7a38-4504-9e2c-272a8ba49167" containerName="payload-processing" containerID="cri-o://e89af969134d09cef09d1db3d5b8702b3f503e05142e99c3307c88e46271e826" gracePeriod=25 Oct 30 01:38:29.167025 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:29.166973 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-19a5f1-5996c85f9d-5j4nj" Oct 30 01:38:29.264617 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:29.264460 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15d60827-7a38-4504-9e2c-272a8ba49167-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15d60827-7a38-4504-9e2c-272a8ba49167-plugins-config-volume\") pod \"15d60827-7a38-4504-9e2c-272a8ba49167\" (UID: \"15d60827-7a38-4504-9e2c-272a8ba49167\") " Oct 30 01:38:29.264826 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:29.264759 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15d60827-7a38-4504-9e2c-272a8ba49167-kube-api-access-mt549\" (UniqueName: \"kubernetes.io/projected/15d60827-7a38-4504-9e2c-272a8ba49167-kube-api-access-mt549\") pod \"15d60827-7a38-4504-9e2c-272a8ba49167\" (UID: \"15d60827-7a38-4504-9e2c-272a8ba49167\") " Oct 30 01:38:29.273065 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:29.273025 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15d60827-7a38-4504-9e2c-272a8ba49167-plugins-config-volume" pod "15d60827-7a38-4504-9e2c-272a8ba49167" (UID: "15d60827-7a38-4504-9e2c-272a8ba49167"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:38:29.288237 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:29.288176 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15d60827-7a38-4504-9e2c-272a8ba49167-kube-api-access-mt549" pod "15d60827-7a38-4504-9e2c-272a8ba49167" (UID: "15d60827-7a38-4504-9e2c-272a8ba49167"). InnerVolumeSpecName "kube-api-access-mt549". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:29.367057 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:29.367016 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt549\" (UniqueName: \"kubernetes.io/projected/15d60827-7a38-4504-9e2c-272a8ba49167-kube-api-access-mt549\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:38:29.367057 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:29.367058 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15d60827-7a38-4504-9e2c-272a8ba49167-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:38:29.593095 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:29.593061 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" Oct 30 01:38:29.593473 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:29.593452 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" Oct 30 01:38:29.594066 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:29.594044 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" Oct 30 01:38:29.715280 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:29.715246 6221 generic.go:356] "Generic (PLEG): container finished" podID="15d60827-7a38-4504-9e2c-272a8ba49167" containerID="e89af969134d09cef09d1db3d5b8702b3f503e05142e99c3307c88e46271e826" exitCode=0 Oct 30 01:38:29.715464 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:29.715306 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-19a5f1-5996c85f9d-5j4nj" Oct 30 01:38:29.715464 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:29.715343 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-19a5f1-5996c85f9d-5j4nj" event={"ID":"15d60827-7a38-4504-9e2c-272a8ba49167","Type":"ContainerDied","Data":"e89af969134d09cef09d1db3d5b8702b3f503e05142e99c3307c88e46271e826"} Oct 30 01:38:29.715464 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:29.715407 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-19a5f1-5996c85f9d-5j4nj" event={"ID":"15d60827-7a38-4504-9e2c-272a8ba49167","Type":"ContainerDied","Data":"d61676fe0c89b3e4ffbfce7246a59626a3da32861cdb53cbb682a0e4ceda6f57"} Oct 30 01:38:29.715464 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:29.715439 6221 scope.go:122] "RemoveContainer" containerID="e89af969134d09cef09d1db3d5b8702b3f503e05142e99c3307c88e46271e826" Oct 30 01:38:29.727916 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:29.727858 6221 scope.go:122] "RemoveContainer" containerID="e89af969134d09cef09d1db3d5b8702b3f503e05142e99c3307c88e46271e826" Oct 30 01:38:29.728282 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:38:29.728255 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e89af969134d09cef09d1db3d5b8702b3f503e05142e99c3307c88e46271e826\": container with ID starting with e89af969134d09cef09d1db3d5b8702b3f503e05142e99c3307c88e46271e826 not found: ID does not exist" containerID="e89af969134d09cef09d1db3d5b8702b3f503e05142e99c3307c88e46271e826" Oct 30 01:38:29.728370 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:29.728297 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e89af969134d09cef09d1db3d5b8702b3f503e05142e99c3307c88e46271e826"} err="failed to get container status \"e89af969134d09cef09d1db3d5b8702b3f503e05142e99c3307c88e46271e826\": rpc error: code = NotFound desc = could not find container \"e89af969134d09cef09d1db3d5b8702b3f503e05142e99c3307c88e46271e826\": container with ID starting with e89af969134d09cef09d1db3d5b8702b3f503e05142e99c3307c88e46271e826 not found: ID does not exist" Oct 30 01:38:29.767806 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:29.767773 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-19a5f1-5996c85f9d-5j4nj"] Oct 30 01:38:29.774898 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:29.774871 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-19a5f1-5996c85f9d-5j4nj"] Oct 30 01:38:30.549305 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:30.549262 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15d60827-7a38-4504-9e2c-272a8ba49167" path="/var/lib/kubelet/pods/15d60827-7a38-4504-9e2c-272a8ba49167/volumes" Oct 30 01:38:32.013673 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:32.013617 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65" Oct 30 01:38:32.130373 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:32.130344 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-workload-socket\") pod \"0dcc7bfb-9764-4e3e-9126-e45fa253a25e\" (UID: \"0dcc7bfb-9764-4e3e-9126-e45fa253a25e\") " Oct 30 01:38:32.130569 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:32.130407 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-istio-data\") pod \"0dcc7bfb-9764-4e3e-9126-e45fa253a25e\" (UID: \"0dcc7bfb-9764-4e3e-9126-e45fa253a25e\") " Oct 30 01:38:32.130569 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:32.130472 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-istiod-ca-cert\") pod \"0dcc7bfb-9764-4e3e-9126-e45fa253a25e\" (UID: \"0dcc7bfb-9764-4e3e-9126-e45fa253a25e\") " Oct 30 01:38:32.130569 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:32.130500 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-istio-envoy\") pod \"0dcc7bfb-9764-4e3e-9126-e45fa253a25e\" (UID: \"0dcc7bfb-9764-4e3e-9126-e45fa253a25e\") " Oct 30 01:38:32.130569 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:32.130527 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-credential-socket\") pod \"0dcc7bfb-9764-4e3e-9126-e45fa253a25e\" (UID: \"0dcc7bfb-9764-4e3e-9126-e45fa253a25e\") " Oct 30 01:38:32.130815 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:32.130735 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-workload-socket" pod "0dcc7bfb-9764-4e3e-9126-e45fa253a25e" (UID: "0dcc7bfb-9764-4e3e-9126-e45fa253a25e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:32.130815 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:32.130750 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-istio-data" pod "0dcc7bfb-9764-4e3e-9126-e45fa253a25e" (UID: "0dcc7bfb-9764-4e3e-9126-e45fa253a25e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:32.130815 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:32.130741 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-workload-certs\") pod \"0dcc7bfb-9764-4e3e-9126-e45fa253a25e\" (UID: \"0dcc7bfb-9764-4e3e-9126-e45fa253a25e\") " Oct 30 01:38:32.130968 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:32.130920 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-workload-certs" pod "0dcc7bfb-9764-4e3e-9126-e45fa253a25e" (UID: "0dcc7bfb-9764-4e3e-9126-e45fa253a25e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:32.130968 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:32.130934 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-kube-api-access-x8ngm\" (UniqueName: \"kubernetes.io/projected/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-kube-api-access-x8ngm\") pod \"0dcc7bfb-9764-4e3e-9126-e45fa253a25e\" (UID: \"0dcc7bfb-9764-4e3e-9126-e45fa253a25e\") " Oct 30 01:38:32.130968 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:32.130939 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-credential-socket" pod "0dcc7bfb-9764-4e3e-9126-e45fa253a25e" (UID: "0dcc7bfb-9764-4e3e-9126-e45fa253a25e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:32.131136 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:32.130978 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-istiod-ca-cert" pod "0dcc7bfb-9764-4e3e-9126-e45fa253a25e" (UID: "0dcc7bfb-9764-4e3e-9126-e45fa253a25e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:38:32.131136 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:32.130988 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-istio-podinfo\") pod \"0dcc7bfb-9764-4e3e-9126-e45fa253a25e\" (UID: \"0dcc7bfb-9764-4e3e-9126-e45fa253a25e\") " Oct 30 01:38:32.131136 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:32.131033 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-istio-token\" (UniqueName: \"kubernetes.io/projected/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-istio-token\") pod \"0dcc7bfb-9764-4e3e-9126-e45fa253a25e\" (UID: \"0dcc7bfb-9764-4e3e-9126-e45fa253a25e\") " Oct 30 01:38:32.131697 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:32.131665 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:38:32.131697 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:32.131689 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:38:32.131866 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:32.131702 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:38:32.131866 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:32.131715 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:38:32.131866 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:32.131729 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:38:32.133134 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:32.133105 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-istio-token" pod "0dcc7bfb-9764-4e3e-9126-e45fa253a25e" (UID: "0dcc7bfb-9764-4e3e-9126-e45fa253a25e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:32.133233 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:32.133134 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-istio-envoy" pod "0dcc7bfb-9764-4e3e-9126-e45fa253a25e" (UID: "0dcc7bfb-9764-4e3e-9126-e45fa253a25e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:38:32.133349 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:32.133325 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-istio-podinfo" pod "0dcc7bfb-9764-4e3e-9126-e45fa253a25e" (UID: "0dcc7bfb-9764-4e3e-9126-e45fa253a25e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:38:32.133469 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:32.133371 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-kube-api-access-x8ngm" pod "0dcc7bfb-9764-4e3e-9126-e45fa253a25e" (UID: "0dcc7bfb-9764-4e3e-9126-e45fa253a25e"). InnerVolumeSpecName "kube-api-access-x8ngm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:32.233014 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:32.232978 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8ngm\" (UniqueName: \"kubernetes.io/projected/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-kube-api-access-x8ngm\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:38:32.233014 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:32.233007 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:38:32.233208 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:32.233022 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:38:32.233208 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:32.233034 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0dcc7bfb-9764-4e3e-9126-e45fa253a25e-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:38:32.824121 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:32.824033 6221 generic.go:356] "Generic (PLEG): container finished" podID="0dcc7bfb-9764-4e3e-9126-e45fa253a25e" containerID="a2853aa6af3dc7068a99b2449661db1691446efcd8074b1e44da68d4b633fa15" exitCode=0 Oct 30 01:38:32.824121 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:32.824093 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65" Oct 30 01:38:32.824305 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:32.824117 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65" event={"ID":"0dcc7bfb-9764-4e3e-9126-e45fa253a25e","Type":"ContainerDied","Data":"a2853aa6af3dc7068a99b2449661db1691446efcd8074b1e44da68d4b633fa15"} Oct 30 01:38:32.824305 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:32.824193 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65" event={"ID":"0dcc7bfb-9764-4e3e-9126-e45fa253a25e","Type":"ContainerDied","Data":"40bfaa59ea7a764f2ef104c8fea59cf658ffd56b9d41f820687fe8d2c5b6b4c7"} Oct 30 01:38:32.824305 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:32.824218 6221 scope.go:122] "RemoveContainer" containerID="a2853aa6af3dc7068a99b2449661db1691446efcd8074b1e44da68d4b633fa15" Oct 30 01:38:32.834141 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:32.834124 6221 scope.go:122] "RemoveContainer" containerID="a2853aa6af3dc7068a99b2449661db1691446efcd8074b1e44da68d4b633fa15" Oct 30 01:38:32.834398 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:38:32.834382 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2853aa6af3dc7068a99b2449661db1691446efcd8074b1e44da68d4b633fa15\": container with ID starting with a2853aa6af3dc7068a99b2449661db1691446efcd8074b1e44da68d4b633fa15 not found: ID does not exist" containerID="a2853aa6af3dc7068a99b2449661db1691446efcd8074b1e44da68d4b633fa15" Oct 30 01:38:32.834444 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:32.834406 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2853aa6af3dc7068a99b2449661db1691446efcd8074b1e44da68d4b633fa15"} err="failed to get container status \"a2853aa6af3dc7068a99b2449661db1691446efcd8074b1e44da68d4b633fa15\": rpc error: code = NotFound desc = could not find container \"a2853aa6af3dc7068a99b2449661db1691446efcd8074b1e44da68d4b633fa15\": container with ID starting with a2853aa6af3dc7068a99b2449661db1691446efcd8074b1e44da68d4b633fa15 not found: ID does not exist" Oct 30 01:38:32.890681 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:32.890652 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65"] Oct 30 01:38:32.902992 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:32.902960 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-bvs65"] Oct 30 01:38:33.328619 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:33.328586 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f7bdcc-7676888b95-hncp6"] Oct 30 01:38:34.626644 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:34.625743 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dcc7bfb-9764-4e3e-9126-e45fa253a25e" path="/var/lib/kubelet/pods/0dcc7bfb-9764-4e3e-9126-e45fa253a25e/volumes" Oct 30 01:38:34.965606 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:34.964741 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:38:38.329737 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:38.329688 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7bdcc-7676888b95-hncp6" podUID="59112cc2-2897-43f7-b2e9-1c01c9f17687" containerName="payload-pre-processing" containerID="cri-o://f81de4590562db58c57319d57452f127a65752228e4a4f21e64e8a414a10b35b" gracePeriod=25 Oct 30 01:38:38.625598 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:38.625571 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7bdcc-7676888b95-hncp6" Oct 30 01:38:38.707943 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:38.707907 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59112cc2-2897-43f7-b2e9-1c01c9f17687-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59112cc2-2897-43f7-b2e9-1c01c9f17687-plugins-config-volume\") pod \"59112cc2-2897-43f7-b2e9-1c01c9f17687\" (UID: \"59112cc2-2897-43f7-b2e9-1c01c9f17687\") " Oct 30 01:38:38.708137 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:38.708078 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59112cc2-2897-43f7-b2e9-1c01c9f17687-kube-api-access-g4mxh\" (UniqueName: \"kubernetes.io/projected/59112cc2-2897-43f7-b2e9-1c01c9f17687-kube-api-access-g4mxh\") pod \"59112cc2-2897-43f7-b2e9-1c01c9f17687\" (UID: \"59112cc2-2897-43f7-b2e9-1c01c9f17687\") " Oct 30 01:38:38.708265 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:38.708243 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59112cc2-2897-43f7-b2e9-1c01c9f17687-plugins-config-volume" pod "59112cc2-2897-43f7-b2e9-1c01c9f17687" (UID: "59112cc2-2897-43f7-b2e9-1c01c9f17687"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:38:38.708576 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:38.708560 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/59112cc2-2897-43f7-b2e9-1c01c9f17687-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:38:38.710119 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:38.710095 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59112cc2-2897-43f7-b2e9-1c01c9f17687-kube-api-access-g4mxh" pod "59112cc2-2897-43f7-b2e9-1c01c9f17687" (UID: "59112cc2-2897-43f7-b2e9-1c01c9f17687"). InnerVolumeSpecName "kube-api-access-g4mxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:38.809512 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:38.809478 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4mxh\" (UniqueName: \"kubernetes.io/projected/59112cc2-2897-43f7-b2e9-1c01c9f17687-kube-api-access-g4mxh\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:38:38.958513 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:38.958427 6221 generic.go:356] "Generic (PLEG): container finished" podID="59112cc2-2897-43f7-b2e9-1c01c9f17687" containerID="f81de4590562db58c57319d57452f127a65752228e4a4f21e64e8a414a10b35b" exitCode=0 Oct 30 01:38:38.958513 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:38.958477 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7bdcc-7676888b95-hncp6" Oct 30 01:38:38.958711 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:38.958517 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7bdcc-7676888b95-hncp6" event={"ID":"59112cc2-2897-43f7-b2e9-1c01c9f17687","Type":"ContainerDied","Data":"f81de4590562db58c57319d57452f127a65752228e4a4f21e64e8a414a10b35b"} Oct 30 01:38:38.958711 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:38.958591 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f7bdcc-7676888b95-hncp6" event={"ID":"59112cc2-2897-43f7-b2e9-1c01c9f17687","Type":"ContainerDied","Data":"c5dbfb1c4817eca3f062b0279fa080f58d7e557c970ed8d5ac5a77e8550efe0d"} Oct 30 01:38:38.958711 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:38.958615 6221 scope.go:122] "RemoveContainer" containerID="f81de4590562db58c57319d57452f127a65752228e4a4f21e64e8a414a10b35b" Oct 30 01:38:38.968098 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:38.968080 6221 scope.go:122] "RemoveContainer" containerID="f81de4590562db58c57319d57452f127a65752228e4a4f21e64e8a414a10b35b" Oct 30 01:38:38.968366 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:38:38.968344 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f81de4590562db58c57319d57452f127a65752228e4a4f21e64e8a414a10b35b\": container with ID starting with f81de4590562db58c57319d57452f127a65752228e4a4f21e64e8a414a10b35b not found: ID does not exist" containerID="f81de4590562db58c57319d57452f127a65752228e4a4f21e64e8a414a10b35b" Oct 30 01:38:38.968434 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:38.968378 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f81de4590562db58c57319d57452f127a65752228e4a4f21e64e8a414a10b35b"} err="failed to get container status \"f81de4590562db58c57319d57452f127a65752228e4a4f21e64e8a414a10b35b\": rpc error: code = NotFound desc = could not find container \"f81de4590562db58c57319d57452f127a65752228e4a4f21e64e8a414a10b35b\": container with ID starting with f81de4590562db58c57319d57452f127a65752228e4a4f21e64e8a414a10b35b not found: ID does not exist" Oct 30 01:38:39.010393 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:39.010363 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f7bdcc-7676888b95-hncp6"] Oct 30 01:38:39.022835 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:39.022814 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f7bdcc-7676888b95-hncp6"] Oct 30 01:38:40.547352 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:40.547321 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59112cc2-2897-43f7-b2e9-1c01c9f17687" path="/var/lib/kubelet/pods/59112cc2-2897-43f7-b2e9-1c01c9f17687/volumes" Oct 30 01:38:51.560798 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:38:51.560752 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:38:54.190413 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:54.190380 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-18560b-7b94fb568-c977d"] Oct 30 01:38:59.195925 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:59.195866 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-18560b-7b94fb568-c977d" podUID="829e28c7-ba53-4e86-afbd-f033698c333d" containerName="payload-processing" containerID="cri-o://8bf31446c67eb0460711618f0a35d5996cb9629b489e0c954f8acb8906fcabd0" gracePeriod=25 Oct 30 01:38:59.570900 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:59.570873 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-18560b-7b94fb568-c977d" Oct 30 01:38:59.647722 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:59.647685 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/829e28c7-ba53-4e86-afbd-f033698c333d-kube-api-access-gtsgv\" (UniqueName: \"kubernetes.io/projected/829e28c7-ba53-4e86-afbd-f033698c333d-kube-api-access-gtsgv\") pod \"829e28c7-ba53-4e86-afbd-f033698c333d\" (UID: \"829e28c7-ba53-4e86-afbd-f033698c333d\") " Oct 30 01:38:59.647895 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:59.647831 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/829e28c7-ba53-4e86-afbd-f033698c333d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/829e28c7-ba53-4e86-afbd-f033698c333d-plugins-config-volume\") pod \"829e28c7-ba53-4e86-afbd-f033698c333d\" (UID: \"829e28c7-ba53-4e86-afbd-f033698c333d\") " Oct 30 01:38:59.648249 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:59.648217 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/829e28c7-ba53-4e86-afbd-f033698c333d-plugins-config-volume" pod "829e28c7-ba53-4e86-afbd-f033698c333d" (UID: "829e28c7-ba53-4e86-afbd-f033698c333d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:38:59.648579 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:59.648557 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/829e28c7-ba53-4e86-afbd-f033698c333d-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:38:59.650060 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:59.650031 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/829e28c7-ba53-4e86-afbd-f033698c333d-kube-api-access-gtsgv" pod "829e28c7-ba53-4e86-afbd-f033698c333d" (UID: "829e28c7-ba53-4e86-afbd-f033698c333d"). InnerVolumeSpecName "kube-api-access-gtsgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:38:59.760674 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:38:59.757557 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtsgv\" (UniqueName: \"kubernetes.io/projected/829e28c7-ba53-4e86-afbd-f033698c333d-kube-api-access-gtsgv\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:39:00.270907 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:00.270866 6221 generic.go:356] "Generic (PLEG): container finished" podID="829e28c7-ba53-4e86-afbd-f033698c333d" containerID="8bf31446c67eb0460711618f0a35d5996cb9629b489e0c954f8acb8906fcabd0" exitCode=0 Oct 30 01:39:00.271580 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:00.271088 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-18560b-7b94fb568-c977d" event={"ID":"829e28c7-ba53-4e86-afbd-f033698c333d","Type":"ContainerDied","Data":"8bf31446c67eb0460711618f0a35d5996cb9629b489e0c954f8acb8906fcabd0"} Oct 30 01:39:00.271580 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:00.271186 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-18560b-7b94fb568-c977d" event={"ID":"829e28c7-ba53-4e86-afbd-f033698c333d","Type":"ContainerDied","Data":"169aa338dbb2db81fa71eed1507eb8087be97ec31872d756c5c0b10882e94215"} Oct 30 01:39:00.271580 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:00.271219 6221 scope.go:122] "RemoveContainer" containerID="8bf31446c67eb0460711618f0a35d5996cb9629b489e0c954f8acb8906fcabd0" Oct 30 01:39:00.271580 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:00.271383 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-18560b-7b94fb568-c977d" Oct 30 01:39:00.303785 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:00.303762 6221 scope.go:122] "RemoveContainer" containerID="8bf31446c67eb0460711618f0a35d5996cb9629b489e0c954f8acb8906fcabd0" Oct 30 01:39:00.304213 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:39:00.304183 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bf31446c67eb0460711618f0a35d5996cb9629b489e0c954f8acb8906fcabd0\": container with ID starting with 8bf31446c67eb0460711618f0a35d5996cb9629b489e0c954f8acb8906fcabd0 not found: ID does not exist" containerID="8bf31446c67eb0460711618f0a35d5996cb9629b489e0c954f8acb8906fcabd0" Oct 30 01:39:00.304324 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:00.304224 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bf31446c67eb0460711618f0a35d5996cb9629b489e0c954f8acb8906fcabd0"} err="failed to get container status \"8bf31446c67eb0460711618f0a35d5996cb9629b489e0c954f8acb8906fcabd0\": rpc error: code = NotFound desc = could not find container \"8bf31446c67eb0460711618f0a35d5996cb9629b489e0c954f8acb8906fcabd0\": container with ID starting with 8bf31446c67eb0460711618f0a35d5996cb9629b489e0c954f8acb8906fcabd0 not found: ID does not exist" Oct 30 01:39:00.343898 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:00.343868 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-18560b-7b94fb568-c977d"] Oct 30 01:39:00.356370 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:00.356338 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-18560b-7b94fb568-c977d"] Oct 30 01:39:00.547436 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:00.547407 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="829e28c7-ba53-4e86-afbd-f033698c333d" path="/var/lib/kubelet/pods/829e28c7-ba53-4e86-afbd-f033698c333d/volumes" Oct 30 01:39:04.025634 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:04.025594 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6b3dbd-6f8f7bfddb-8t572"] Oct 30 01:39:04.061186 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:04.061153 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3dbd-775fc65d4c-bjhqz"] Oct 30 01:39:06.035664 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:06.035635 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:39:09.026027 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:09.025920 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3dbd-6f8f7bfddb-8t572" podUID="69f4a147-db8e-4b2c-a3a1-c7259e37ceba" containerName="payload-processing" containerID="cri-o://80cde8c6a61690e8173225a459389407d57a97d5dc446cc7c1c5b7b658572c8d" gracePeriod=25 Oct 30 01:39:09.062334 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:09.062284 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3dbd-775fc65d4c-bjhqz" podUID="a8ac61bc-e5e8-4615-9767-9991ba366b7a" containerName="payload-pre-processing" containerID="cri-o://fcd9f04ab19f8d3f06f52bb166ff5ee26a7f1250bfa0072ca5c70a3fc6a2231c" gracePeriod=25 Oct 30 01:39:09.337323 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:09.337287 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3dbd-6f8f7bfddb-8t572" Oct 30 01:39:09.371185 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:09.371154 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69f4a147-db8e-4b2c-a3a1-c7259e37ceba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69f4a147-db8e-4b2c-a3a1-c7259e37ceba-plugins-config-volume\") pod \"69f4a147-db8e-4b2c-a3a1-c7259e37ceba\" (UID: \"69f4a147-db8e-4b2c-a3a1-c7259e37ceba\") " Oct 30 01:39:09.371362 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:09.371350 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69f4a147-db8e-4b2c-a3a1-c7259e37ceba-kube-api-access-px8w6\" (UniqueName: \"kubernetes.io/projected/69f4a147-db8e-4b2c-a3a1-c7259e37ceba-kube-api-access-px8w6\") pod \"69f4a147-db8e-4b2c-a3a1-c7259e37ceba\" (UID: \"69f4a147-db8e-4b2c-a3a1-c7259e37ceba\") " Oct 30 01:39:09.371572 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:09.371546 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69f4a147-db8e-4b2c-a3a1-c7259e37ceba-plugins-config-volume" pod "69f4a147-db8e-4b2c-a3a1-c7259e37ceba" (UID: "69f4a147-db8e-4b2c-a3a1-c7259e37ceba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:39:09.371818 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:09.371802 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69f4a147-db8e-4b2c-a3a1-c7259e37ceba-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:39:09.373402 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:09.373382 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69f4a147-db8e-4b2c-a3a1-c7259e37ceba-kube-api-access-px8w6" pod "69f4a147-db8e-4b2c-a3a1-c7259e37ceba" (UID: "69f4a147-db8e-4b2c-a3a1-c7259e37ceba"). InnerVolumeSpecName "kube-api-access-px8w6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:39:09.395248 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:09.395222 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3dbd-775fc65d4c-bjhqz" Oct 30 01:39:09.472435 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:09.472404 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8ac61bc-e5e8-4615-9767-9991ba366b7a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8ac61bc-e5e8-4615-9767-9991ba366b7a-plugins-config-volume\") pod \"a8ac61bc-e5e8-4615-9767-9991ba366b7a\" (UID: \"a8ac61bc-e5e8-4615-9767-9991ba366b7a\") " Oct 30 01:39:09.472672 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:09.472558 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8ac61bc-e5e8-4615-9767-9991ba366b7a-kube-api-access-qz74m\" (UniqueName: \"kubernetes.io/projected/a8ac61bc-e5e8-4615-9767-9991ba366b7a-kube-api-access-qz74m\") pod \"a8ac61bc-e5e8-4615-9767-9991ba366b7a\" (UID: \"a8ac61bc-e5e8-4615-9767-9991ba366b7a\") " Oct 30 01:39:09.472921 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:09.472890 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8ac61bc-e5e8-4615-9767-9991ba366b7a-plugins-config-volume" pod "a8ac61bc-e5e8-4615-9767-9991ba366b7a" (UID: "a8ac61bc-e5e8-4615-9767-9991ba366b7a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:39:09.473094 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:09.473074 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px8w6\" (UniqueName: \"kubernetes.io/projected/69f4a147-db8e-4b2c-a3a1-c7259e37ceba-kube-api-access-px8w6\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:39:09.473179 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:09.473098 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8ac61bc-e5e8-4615-9767-9991ba366b7a-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:39:09.474742 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:09.474717 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8ac61bc-e5e8-4615-9767-9991ba366b7a-kube-api-access-qz74m" pod "a8ac61bc-e5e8-4615-9767-9991ba366b7a" (UID: "a8ac61bc-e5e8-4615-9767-9991ba366b7a"). InnerVolumeSpecName "kube-api-access-qz74m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:39:09.575056 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:09.574968 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz74m\" (UniqueName: \"kubernetes.io/projected/a8ac61bc-e5e8-4615-9767-9991ba366b7a-kube-api-access-qz74m\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:39:09.677373 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:09.677330 6221 generic.go:356] "Generic (PLEG): container finished" podID="a8ac61bc-e5e8-4615-9767-9991ba366b7a" containerID="fcd9f04ab19f8d3f06f52bb166ff5ee26a7f1250bfa0072ca5c70a3fc6a2231c" exitCode=0 Oct 30 01:39:09.678037 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:09.677700 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3dbd-775fc65d4c-bjhqz" Oct 30 01:39:09.678959 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:09.677723 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3dbd-775fc65d4c-bjhqz" event={"ID":"a8ac61bc-e5e8-4615-9767-9991ba366b7a","Type":"ContainerDied","Data":"fcd9f04ab19f8d3f06f52bb166ff5ee26a7f1250bfa0072ca5c70a3fc6a2231c"} Oct 30 01:39:09.678959 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:09.678480 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3dbd-775fc65d4c-bjhqz" event={"ID":"a8ac61bc-e5e8-4615-9767-9991ba366b7a","Type":"ContainerDied","Data":"13dec5d9d8fff04490e4f04e399c02c6ab48500fcc473ce696cd92429b874ce0"} Oct 30 01:39:09.678959 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:09.678514 6221 scope.go:122] "RemoveContainer" containerID="fcd9f04ab19f8d3f06f52bb166ff5ee26a7f1250bfa0072ca5c70a3fc6a2231c" Oct 30 01:39:09.696440 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:09.696399 6221 generic.go:356] "Generic (PLEG): container finished" podID="69f4a147-db8e-4b2c-a3a1-c7259e37ceba" containerID="80cde8c6a61690e8173225a459389407d57a97d5dc446cc7c1c5b7b658572c8d" exitCode=0 Oct 30 01:39:09.696610 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:09.696461 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3dbd-6f8f7bfddb-8t572" event={"ID":"69f4a147-db8e-4b2c-a3a1-c7259e37ceba","Type":"ContainerDied","Data":"80cde8c6a61690e8173225a459389407d57a97d5dc446cc7c1c5b7b658572c8d"} Oct 30 01:39:09.696610 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:09.696507 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3dbd-6f8f7bfddb-8t572" event={"ID":"69f4a147-db8e-4b2c-a3a1-c7259e37ceba","Type":"ContainerDied","Data":"9c097cf76bf46d5e6da3c1098cb038309a4c56c142a244a210f4328b45a7a0bc"} Oct 30 01:39:09.696610 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:09.696602 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-6b3dbd-6f8f7bfddb-8t572" Oct 30 01:39:09.711113 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:09.711085 6221 scope.go:122] "RemoveContainer" containerID="fcd9f04ab19f8d3f06f52bb166ff5ee26a7f1250bfa0072ca5c70a3fc6a2231c" Oct 30 01:39:09.711482 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:39:09.711457 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcd9f04ab19f8d3f06f52bb166ff5ee26a7f1250bfa0072ca5c70a3fc6a2231c\": container with ID starting with fcd9f04ab19f8d3f06f52bb166ff5ee26a7f1250bfa0072ca5c70a3fc6a2231c not found: ID does not exist" containerID="fcd9f04ab19f8d3f06f52bb166ff5ee26a7f1250bfa0072ca5c70a3fc6a2231c" Oct 30 01:39:09.711624 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:09.711496 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcd9f04ab19f8d3f06f52bb166ff5ee26a7f1250bfa0072ca5c70a3fc6a2231c"} err="failed to get container status \"fcd9f04ab19f8d3f06f52bb166ff5ee26a7f1250bfa0072ca5c70a3fc6a2231c\": rpc error: code = NotFound desc = could not find container \"fcd9f04ab19f8d3f06f52bb166ff5ee26a7f1250bfa0072ca5c70a3fc6a2231c\": container with ID starting with fcd9f04ab19f8d3f06f52bb166ff5ee26a7f1250bfa0072ca5c70a3fc6a2231c not found: ID does not exist" Oct 30 01:39:09.711624 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:09.711521 6221 scope.go:122] "RemoveContainer" containerID="80cde8c6a61690e8173225a459389407d57a97d5dc446cc7c1c5b7b658572c8d" Oct 30 01:39:09.721890 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:09.721871 6221 scope.go:122] "RemoveContainer" containerID="80cde8c6a61690e8173225a459389407d57a97d5dc446cc7c1c5b7b658572c8d" Oct 30 01:39:09.722207 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:39:09.722184 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80cde8c6a61690e8173225a459389407d57a97d5dc446cc7c1c5b7b658572c8d\": container with ID starting with 80cde8c6a61690e8173225a459389407d57a97d5dc446cc7c1c5b7b658572c8d not found: ID does not exist" containerID="80cde8c6a61690e8173225a459389407d57a97d5dc446cc7c1c5b7b658572c8d" Oct 30 01:39:09.722281 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:09.722220 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80cde8c6a61690e8173225a459389407d57a97d5dc446cc7c1c5b7b658572c8d"} err="failed to get container status \"80cde8c6a61690e8173225a459389407d57a97d5dc446cc7c1c5b7b658572c8d\": rpc error: code = NotFound desc = could not find container \"80cde8c6a61690e8173225a459389407d57a97d5dc446cc7c1c5b7b658572c8d\": container with ID starting with 80cde8c6a61690e8173225a459389407d57a97d5dc446cc7c1c5b7b658572c8d not found: ID does not exist" Oct 30 01:39:09.753858 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:09.753823 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6b3dbd-6f8f7bfddb-8t572"] Oct 30 01:39:09.757113 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:09.757076 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-6b3dbd-6f8f7bfddb-8t572"] Oct 30 01:39:09.791698 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:09.791667 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3dbd-775fc65d4c-bjhqz"] Oct 30 01:39:09.794276 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:09.794253 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-6b3dbd-775fc65d4c-bjhqz"] Oct 30 01:39:10.553900 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:10.553863 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69f4a147-db8e-4b2c-a3a1-c7259e37ceba" path="/var/lib/kubelet/pods/69f4a147-db8e-4b2c-a3a1-c7259e37ceba/volumes" Oct 30 01:39:10.554311 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:10.554297 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8ac61bc-e5e8-4615-9767-9991ba366b7a" path="/var/lib/kubelet/pods/a8ac61bc-e5e8-4615-9767-9991ba366b7a/volumes" Oct 30 01:39:21.494739 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:39:21.494706 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:39:21.821269 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:21.821226 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0db534-7f9675fdc5-jr2qx"] Oct 30 01:39:21.821834 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:21.821819 6221 cpu_manager.go:414] "containerMap: removing container" podUID="0dcc7bfb-9764-4e3e-9126-e45fa253a25e" containerName="istio-proxy" Oct 30 01:39:21.821904 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:21.821835 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dcc7bfb-9764-4e3e-9126-e45fa253a25e" containerName="istio-proxy" Oct 30 01:39:21.821904 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:21.821851 6221 cpu_manager.go:414] "containerMap: removing container" podUID="a8ac61bc-e5e8-4615-9767-9991ba366b7a" containerName="payload-pre-processing" Oct 30 01:39:21.821904 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:21.821856 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8ac61bc-e5e8-4615-9767-9991ba366b7a" containerName="payload-pre-processing" Oct 30 01:39:21.821904 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:21.821869 6221 cpu_manager.go:414] "containerMap: removing container" podUID="829e28c7-ba53-4e86-afbd-f033698c333d" containerName="payload-processing" Oct 30 01:39:21.821904 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:21.821873 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="829e28c7-ba53-4e86-afbd-f033698c333d" containerName="payload-processing" Oct 30 01:39:21.821904 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:21.821879 6221 cpu_manager.go:414] "containerMap: removing container" podUID="59112cc2-2897-43f7-b2e9-1c01c9f17687" containerName="payload-pre-processing" Oct 30 01:39:21.821904 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:21.821886 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59112cc2-2897-43f7-b2e9-1c01c9f17687" containerName="payload-pre-processing" Oct 30 01:39:21.821904 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:21.821902 6221 cpu_manager.go:414] "containerMap: removing container" podUID="15d60827-7a38-4504-9e2c-272a8ba49167" containerName="payload-processing" Oct 30 01:39:21.821904 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:21.821907 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15d60827-7a38-4504-9e2c-272a8ba49167" containerName="payload-processing" Oct 30 01:39:21.822162 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:21.821931 6221 cpu_manager.go:414] "containerMap: removing container" podUID="69f4a147-db8e-4b2c-a3a1-c7259e37ceba" containerName="payload-processing" Oct 30 01:39:21.822162 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:21.821936 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69f4a147-db8e-4b2c-a3a1-c7259e37ceba" containerName="payload-processing" Oct 30 01:39:21.822162 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:21.822013 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="15d60827-7a38-4504-9e2c-272a8ba49167" containerName="payload-processing" Oct 30 01:39:21.822162 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:21.822025 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="69f4a147-db8e-4b2c-a3a1-c7259e37ceba" containerName="payload-processing" Oct 30 01:39:21.822162 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:21.822033 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dcc7bfb-9764-4e3e-9126-e45fa253a25e" containerName="istio-proxy" Oct 30 01:39:21.822162 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:21.822039 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8ac61bc-e5e8-4615-9767-9991ba366b7a" containerName="payload-pre-processing" Oct 30 01:39:21.822162 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:21.822048 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="829e28c7-ba53-4e86-afbd-f033698c333d" containerName="payload-processing" Oct 30 01:39:21.822162 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:21.822054 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="59112cc2-2897-43f7-b2e9-1c01c9f17687" containerName="payload-pre-processing" Oct 30 01:39:21.825608 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:21.825581 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db534-7f9675fdc5-jr2qx" Oct 30 01:39:21.855165 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:21.855135 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0db534-7f9675fdc5-jr2qx"] Oct 30 01:39:21.880032 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:21.880002 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-0db534\"" Oct 30 01:39:21.926606 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:21.926579 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-0db534-dockercfg-mxspk\"" Oct 30 01:39:21.940118 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:21.940086 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnc2c\" (UniqueName: \"kubernetes.io/projected/b2f52c71-1aad-4817-b9f8-9fd8759f1961-kube-api-access-rnc2c\") pod \"payload-pre-processing-e2e-ipp-b-0db534-7f9675fdc5-jr2qx\" (UID: \"b2f52c71-1aad-4817-b9f8-9fd8759f1961\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db534-7f9675fdc5-jr2qx" Oct 30 01:39:21.940372 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:21.940353 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2f52c71-1aad-4817-b9f8-9fd8759f1961-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0db534-7f9675fdc5-jr2qx\" (UID: \"b2f52c71-1aad-4817-b9f8-9fd8759f1961\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db534-7f9675fdc5-jr2qx" Oct 30 01:39:22.056200 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:22.044664 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2f52c71-1aad-4817-b9f8-9fd8759f1961-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0db534-7f9675fdc5-jr2qx\" (UID: \"b2f52c71-1aad-4817-b9f8-9fd8759f1961\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db534-7f9675fdc5-jr2qx" Oct 30 01:39:22.056200 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:22.044789 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnc2c\" (UniqueName: \"kubernetes.io/projected/b2f52c71-1aad-4817-b9f8-9fd8759f1961-kube-api-access-rnc2c\") pod \"payload-pre-processing-e2e-ipp-b-0db534-7f9675fdc5-jr2qx\" (UID: \"b2f52c71-1aad-4817-b9f8-9fd8759f1961\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db534-7f9675fdc5-jr2qx" Oct 30 01:39:22.056200 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:22.046434 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2f52c71-1aad-4817-b9f8-9fd8759f1961-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0db534-7f9675fdc5-jr2qx\" (UID: \"b2f52c71-1aad-4817-b9f8-9fd8759f1961\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db534-7f9675fdc5-jr2qx" Oct 30 01:39:22.083631 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:22.078285 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnc2c\" (UniqueName: \"kubernetes.io/projected/b2f52c71-1aad-4817-b9f8-9fd8759f1961-kube-api-access-rnc2c\") pod \"payload-pre-processing-e2e-ipp-b-0db534-7f9675fdc5-jr2qx\" (UID: \"b2f52c71-1aad-4817-b9f8-9fd8759f1961\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db534-7f9675fdc5-jr2qx" Oct 30 01:39:22.143883 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:22.143836 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db534-7f9675fdc5-jr2qx" Oct 30 01:39:22.618067 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:22.618039 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0db534-7f9675fdc5-jr2qx"] Oct 30 01:39:22.618950 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:39:22.618923 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2f52c71_1aad_4817_b9f8_9fd8759f1961.slice/crio-e31b95caca84e2efdf7d20a819730f7fec2a0577adc375b2eef15d1f28502602 WatchSource:0}: Error finding container e31b95caca84e2efdf7d20a819730f7fec2a0577adc375b2eef15d1f28502602: Status 404 returned error can't find the container with id e31b95caca84e2efdf7d20a819730f7fec2a0577adc375b2eef15d1f28502602 Oct 30 01:39:22.825065 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:22.825025 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db534-7f9675fdc5-jr2qx" event={"ID":"b2f52c71-1aad-4817-b9f8-9fd8759f1961","Type":"ContainerStarted","Data":"177ae38f1caa01b7ccc87d3bd1f049219268dc05ba08b06f9c68059daa269c86"} Oct 30 01:39:22.825212 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:22.825084 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db534-7f9675fdc5-jr2qx" event={"ID":"b2f52c71-1aad-4817-b9f8-9fd8759f1961","Type":"ContainerStarted","Data":"e31b95caca84e2efdf7d20a819730f7fec2a0577adc375b2eef15d1f28502602"} Oct 30 01:39:22.825212 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:22.825177 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db534-7f9675fdc5-jr2qx" Oct 30 01:39:22.866579 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:22.866474 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db534-7f9675fdc5-jr2qx" podStartSLOduration=1.8664599 podStartE2EDuration="1.8664599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:39:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:39:22.863258505 +0000 UTC m=+1286.441545842" watchObservedRunningTime="2026-10-30 01:39:22.8664599 +0000 UTC m=+1286.444747235" Oct 30 01:39:33.834197 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:33.834168 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db534-7f9675fdc5-jr2qx" Oct 30 01:39:34.221516 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:34.221436 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb"] Oct 30 01:39:34.221712 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:34.221651 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb" podUID="0d56da67-6cfc-49c4-802b-3c9579c87110" containerName="istio-proxy" containerID="cri-o://cdc620cdba9036461218c8eaf7939e0e3a185372e9ba03380aeac0cbfed1e20f" gracePeriod=30 Oct 30 01:39:34.233966 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:34.233931 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6"] Oct 30 01:39:34.234207 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:34.234181 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6" podUID="b5499971-7479-4b2e-969c-e10ab43b95f1" containerName="istio-proxy" containerID="cri-o://5f1ed4126b73545b5f734a0337a9383cfb6471dfff05ffda627d642e9db9de1d" gracePeriod=30 Oct 30 01:39:34.236636 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:34.236606 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq"] Oct 30 01:39:34.237088 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:34.237050 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq" podUID="6858a801-456a-4716-be05-8a3a7bd80334" containerName="istio-proxy" containerID="cri-o://3ab22f87b33c77d11f6f9ded0e2b60d140bdfb674713ad874369a98eaa5a0d72" gracePeriod=30 Oct 30 01:39:34.239715 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:34.239685 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb"] Oct 30 01:39:34.240058 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:34.239962 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb" podUID="0ef9c920-60ba-43ce-a4a2-2f5bc2731279" containerName="istio-proxy" containerID="cri-o://25a46734f3ed22e9662fc8e7fdf7d563f4c419cccfa061acdc1347852a075f2b" gracePeriod=30 Oct 30 01:39:39.587133 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.587096 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb" Oct 30 01:39:39.598987 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.598964 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6" Oct 30 01:39:39.604423 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.604398 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq" Oct 30 01:39:39.609285 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.609266 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb" Oct 30 01:39:39.662556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.662498 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-workload-certs\") pod \"0ef9c920-60ba-43ce-a4a2-2f5bc2731279\" (UID: \"0ef9c920-60ba-43ce-a4a2-2f5bc2731279\") " Oct 30 01:39:39.662738 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.662629 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-istio-data\") pod \"0ef9c920-60ba-43ce-a4a2-2f5bc2731279\" (UID: \"0ef9c920-60ba-43ce-a4a2-2f5bc2731279\") " Oct 30 01:39:39.662738 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.662664 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d56da67-6cfc-49c4-802b-3c9579c87110-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d56da67-6cfc-49c4-802b-3c9579c87110-credential-socket\") pod \"0d56da67-6cfc-49c4-802b-3c9579c87110\" (UID: \"0d56da67-6cfc-49c4-802b-3c9579c87110\") " Oct 30 01:39:39.662738 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.662701 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6858a801-456a-4716-be05-8a3a7bd80334-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6858a801-456a-4716-be05-8a3a7bd80334-istiod-ca-cert\") pod \"6858a801-456a-4716-be05-8a3a7bd80334\" (UID: \"6858a801-456a-4716-be05-8a3a7bd80334\") " Oct 30 01:39:39.662738 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.662732 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5499971-7479-4b2e-969c-e10ab43b95f1-kube-api-access-qd7zh\" (UniqueName: \"kubernetes.io/projected/b5499971-7479-4b2e-969c-e10ab43b95f1-kube-api-access-qd7zh\") pod \"b5499971-7479-4b2e-969c-e10ab43b95f1\" (UID: \"b5499971-7479-4b2e-969c-e10ab43b95f1\") " Oct 30 01:39:39.662956 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.662764 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5499971-7479-4b2e-969c-e10ab43b95f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5499971-7479-4b2e-969c-e10ab43b95f1-istio-data\") pod \"b5499971-7479-4b2e-969c-e10ab43b95f1\" (UID: \"b5499971-7479-4b2e-969c-e10ab43b95f1\") " Oct 30 01:39:39.662956 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.662805 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6858a801-456a-4716-be05-8a3a7bd80334-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6858a801-456a-4716-be05-8a3a7bd80334-istio-envoy\") pod \"6858a801-456a-4716-be05-8a3a7bd80334\" (UID: \"6858a801-456a-4716-be05-8a3a7bd80334\") " Oct 30 01:39:39.662956 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.662794 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-workload-certs" pod "0ef9c920-60ba-43ce-a4a2-2f5bc2731279" (UID: "0ef9c920-60ba-43ce-a4a2-2f5bc2731279"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:39:39.662956 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.662837 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5499971-7479-4b2e-969c-e10ab43b95f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5499971-7479-4b2e-969c-e10ab43b95f1-credential-socket\") pod \"b5499971-7479-4b2e-969c-e10ab43b95f1\" (UID: \"b5499971-7479-4b2e-969c-e10ab43b95f1\") " Oct 30 01:39:39.662956 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.662849 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-istio-data" pod "0ef9c920-60ba-43ce-a4a2-2f5bc2731279" (UID: "0ef9c920-60ba-43ce-a4a2-2f5bc2731279"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:39:39.662956 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.662878 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6858a801-456a-4716-be05-8a3a7bd80334-kube-api-access-zwn7g\" (UniqueName: \"kubernetes.io/projected/6858a801-456a-4716-be05-8a3a7bd80334-kube-api-access-zwn7g\") pod \"6858a801-456a-4716-be05-8a3a7bd80334\" (UID: \"6858a801-456a-4716-be05-8a3a7bd80334\") " Oct 30 01:39:39.662956 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.662908 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5499971-7479-4b2e-969c-e10ab43b95f1-istio-token\" (UniqueName: \"kubernetes.io/projected/b5499971-7479-4b2e-969c-e10ab43b95f1-istio-token\") pod \"b5499971-7479-4b2e-969c-e10ab43b95f1\" (UID: \"b5499971-7479-4b2e-969c-e10ab43b95f1\") " Oct 30 01:39:39.662956 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.662941 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6858a801-456a-4716-be05-8a3a7bd80334-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6858a801-456a-4716-be05-8a3a7bd80334-workload-certs\") pod \"6858a801-456a-4716-be05-8a3a7bd80334\" (UID: \"6858a801-456a-4716-be05-8a3a7bd80334\") " Oct 30 01:39:39.663358 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.662973 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d56da67-6cfc-49c4-802b-3c9579c87110-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d56da67-6cfc-49c4-802b-3c9579c87110-istio-envoy\") pod \"0d56da67-6cfc-49c4-802b-3c9579c87110\" (UID: \"0d56da67-6cfc-49c4-802b-3c9579c87110\") " Oct 30 01:39:39.663358 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.662999 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6858a801-456a-4716-be05-8a3a7bd80334-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6858a801-456a-4716-be05-8a3a7bd80334-workload-socket\") pod \"6858a801-456a-4716-be05-8a3a7bd80334\" (UID: \"6858a801-456a-4716-be05-8a3a7bd80334\") " Oct 30 01:39:39.663358 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.663007 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5499971-7479-4b2e-969c-e10ab43b95f1-istio-data" pod "b5499971-7479-4b2e-969c-e10ab43b95f1" (UID: "b5499971-7479-4b2e-969c-e10ab43b95f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:39:39.663358 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.663022 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-istio-token\" (UniqueName: \"kubernetes.io/projected/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-istio-token\") pod \"0ef9c920-60ba-43ce-a4a2-2f5bc2731279\" (UID: \"0ef9c920-60ba-43ce-a4a2-2f5bc2731279\") " Oct 30 01:39:39.663358 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.663040 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6858a801-456a-4716-be05-8a3a7bd80334-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6858a801-456a-4716-be05-8a3a7bd80334-istio-podinfo\") pod \"6858a801-456a-4716-be05-8a3a7bd80334\" (UID: \"6858a801-456a-4716-be05-8a3a7bd80334\") " Oct 30 01:39:39.663358 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.663062 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5499971-7479-4b2e-969c-e10ab43b95f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5499971-7479-4b2e-969c-e10ab43b95f1-workload-certs\") pod \"b5499971-7479-4b2e-969c-e10ab43b95f1\" (UID: \"b5499971-7479-4b2e-969c-e10ab43b95f1\") " Oct 30 01:39:39.663358 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.663114 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5499971-7479-4b2e-969c-e10ab43b95f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5499971-7479-4b2e-969c-e10ab43b95f1-istio-envoy\") pod \"b5499971-7479-4b2e-969c-e10ab43b95f1\" (UID: \"b5499971-7479-4b2e-969c-e10ab43b95f1\") " Oct 30 01:39:39.663358 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.663133 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5499971-7479-4b2e-969c-e10ab43b95f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5499971-7479-4b2e-969c-e10ab43b95f1-workload-socket\") pod \"b5499971-7479-4b2e-969c-e10ab43b95f1\" (UID: \"b5499971-7479-4b2e-969c-e10ab43b95f1\") " Oct 30 01:39:39.663358 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.663169 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-istio-podinfo\") pod \"0ef9c920-60ba-43ce-a4a2-2f5bc2731279\" (UID: \"0ef9c920-60ba-43ce-a4a2-2f5bc2731279\") " Oct 30 01:39:39.663358 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.663221 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-kube-api-access-qfv65\" (UniqueName: \"kubernetes.io/projected/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-kube-api-access-qfv65\") pod \"0ef9c920-60ba-43ce-a4a2-2f5bc2731279\" (UID: \"0ef9c920-60ba-43ce-a4a2-2f5bc2731279\") " Oct 30 01:39:39.663358 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.663250 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6858a801-456a-4716-be05-8a3a7bd80334-istio-token\" (UniqueName: \"kubernetes.io/projected/6858a801-456a-4716-be05-8a3a7bd80334-istio-token\") pod \"6858a801-456a-4716-be05-8a3a7bd80334\" (UID: \"6858a801-456a-4716-be05-8a3a7bd80334\") " Oct 30 01:39:39.663358 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.663329 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d56da67-6cfc-49c4-802b-3c9579c87110-istio-token\" (UniqueName: \"kubernetes.io/projected/0d56da67-6cfc-49c4-802b-3c9579c87110-istio-token\") pod \"0d56da67-6cfc-49c4-802b-3c9579c87110\" (UID: \"0d56da67-6cfc-49c4-802b-3c9579c87110\") " Oct 30 01:39:39.663993 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.663394 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6858a801-456a-4716-be05-8a3a7bd80334-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6858a801-456a-4716-be05-8a3a7bd80334-credential-socket\") pod \"6858a801-456a-4716-be05-8a3a7bd80334\" (UID: \"6858a801-456a-4716-be05-8a3a7bd80334\") " Oct 30 01:39:39.663993 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.663447 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5499971-7479-4b2e-969c-e10ab43b95f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5499971-7479-4b2e-969c-e10ab43b95f1-istio-podinfo\") pod \"b5499971-7479-4b2e-969c-e10ab43b95f1\" (UID: \"b5499971-7479-4b2e-969c-e10ab43b95f1\") " Oct 30 01:39:39.663993 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.663493 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d56da67-6cfc-49c4-802b-3c9579c87110-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d56da67-6cfc-49c4-802b-3c9579c87110-workload-certs\") pod \"0d56da67-6cfc-49c4-802b-3c9579c87110\" (UID: \"0d56da67-6cfc-49c4-802b-3c9579c87110\") " Oct 30 01:39:39.663993 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.663584 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d56da67-6cfc-49c4-802b-3c9579c87110-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d56da67-6cfc-49c4-802b-3c9579c87110-istiod-ca-cert\") pod \"0d56da67-6cfc-49c4-802b-3c9579c87110\" (UID: \"0d56da67-6cfc-49c4-802b-3c9579c87110\") " Oct 30 01:39:39.663993 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.663610 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d56da67-6cfc-49c4-802b-3c9579c87110-kube-api-access-zqdqw\" (UniqueName: \"kubernetes.io/projected/0d56da67-6cfc-49c4-802b-3c9579c87110-kube-api-access-zqdqw\") pod \"0d56da67-6cfc-49c4-802b-3c9579c87110\" (UID: \"0d56da67-6cfc-49c4-802b-3c9579c87110\") " Oct 30 01:39:39.663993 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.663631 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6858a801-456a-4716-be05-8a3a7bd80334-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6858a801-456a-4716-be05-8a3a7bd80334-istio-data\") pod \"6858a801-456a-4716-be05-8a3a7bd80334\" (UID: \"6858a801-456a-4716-be05-8a3a7bd80334\") " Oct 30 01:39:39.663993 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.663650 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5499971-7479-4b2e-969c-e10ab43b95f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5499971-7479-4b2e-969c-e10ab43b95f1-istiod-ca-cert\") pod \"b5499971-7479-4b2e-969c-e10ab43b95f1\" (UID: \"b5499971-7479-4b2e-969c-e10ab43b95f1\") " Oct 30 01:39:39.663993 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.663680 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-istio-envoy\") pod \"0ef9c920-60ba-43ce-a4a2-2f5bc2731279\" (UID: \"0ef9c920-60ba-43ce-a4a2-2f5bc2731279\") " Oct 30 01:39:39.663993 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.663716 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-istiod-ca-cert\") pod \"0ef9c920-60ba-43ce-a4a2-2f5bc2731279\" (UID: \"0ef9c920-60ba-43ce-a4a2-2f5bc2731279\") " Oct 30 01:39:39.663993 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.663742 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-credential-socket\") pod \"0ef9c920-60ba-43ce-a4a2-2f5bc2731279\" (UID: \"0ef9c920-60ba-43ce-a4a2-2f5bc2731279\") " Oct 30 01:39:39.663993 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.663763 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d56da67-6cfc-49c4-802b-3c9579c87110-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d56da67-6cfc-49c4-802b-3c9579c87110-istio-data\") pod \"0d56da67-6cfc-49c4-802b-3c9579c87110\" (UID: \"0d56da67-6cfc-49c4-802b-3c9579c87110\") " Oct 30 01:39:39.663993 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.663791 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d56da67-6cfc-49c4-802b-3c9579c87110-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d56da67-6cfc-49c4-802b-3c9579c87110-workload-socket\") pod \"0d56da67-6cfc-49c4-802b-3c9579c87110\" (UID: \"0d56da67-6cfc-49c4-802b-3c9579c87110\") " Oct 30 01:39:39.663993 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.663834 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5499971-7479-4b2e-969c-e10ab43b95f1-workload-socket" pod "b5499971-7479-4b2e-969c-e10ab43b95f1" (UID: "b5499971-7479-4b2e-969c-e10ab43b95f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:39:39.663993 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.663857 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d56da67-6cfc-49c4-802b-3c9579c87110-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d56da67-6cfc-49c4-802b-3c9579c87110-istio-podinfo\") pod \"0d56da67-6cfc-49c4-802b-3c9579c87110\" (UID: \"0d56da67-6cfc-49c4-802b-3c9579c87110\") " Oct 30 01:39:39.664743 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.664117 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-workload-socket\") pod \"0ef9c920-60ba-43ce-a4a2-2f5bc2731279\" (UID: \"0ef9c920-60ba-43ce-a4a2-2f5bc2731279\") " Oct 30 01:39:39.666670 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.664994 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5499971-7479-4b2e-969c-e10ab43b95f1-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:39:39.666670 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.665024 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:39:39.666670 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.665039 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:39:39.666670 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.665055 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5499971-7479-4b2e-969c-e10ab43b95f1-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:39:39.666670 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.665190 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6858a801-456a-4716-be05-8a3a7bd80334-istiod-ca-cert" pod "6858a801-456a-4716-be05-8a3a7bd80334" (UID: "6858a801-456a-4716-be05-8a3a7bd80334"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:39:39.666670 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.665519 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5499971-7479-4b2e-969c-e10ab43b95f1-kube-api-access-qd7zh" pod "b5499971-7479-4b2e-969c-e10ab43b95f1" (UID: "b5499971-7479-4b2e-969c-e10ab43b95f1"). InnerVolumeSpecName "kube-api-access-qd7zh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:39:39.666670 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.665573 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d56da67-6cfc-49c4-802b-3c9579c87110-credential-socket" pod "0d56da67-6cfc-49c4-802b-3c9579c87110" (UID: "0d56da67-6cfc-49c4-802b-3c9579c87110"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:39:39.666670 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.665762 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5499971-7479-4b2e-969c-e10ab43b95f1-credential-socket" pod "b5499971-7479-4b2e-969c-e10ab43b95f1" (UID: "b5499971-7479-4b2e-969c-e10ab43b95f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:39:39.666670 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.665874 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d56da67-6cfc-49c4-802b-3c9579c87110-istiod-ca-cert" pod "0d56da67-6cfc-49c4-802b-3c9579c87110" (UID: "0d56da67-6cfc-49c4-802b-3c9579c87110"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:39:39.666670 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.666138 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6858a801-456a-4716-be05-8a3a7bd80334-istio-podinfo" pod "6858a801-456a-4716-be05-8a3a7bd80334" (UID: "6858a801-456a-4716-be05-8a3a7bd80334"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:39:39.666670 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.666192 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-workload-socket" pod "0ef9c920-60ba-43ce-a4a2-2f5bc2731279" (UID: "0ef9c920-60ba-43ce-a4a2-2f5bc2731279"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:39:39.666670 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.666354 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6858a801-456a-4716-be05-8a3a7bd80334-workload-socket" pod "6858a801-456a-4716-be05-8a3a7bd80334" (UID: "6858a801-456a-4716-be05-8a3a7bd80334"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:39:39.666670 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.666390 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5499971-7479-4b2e-969c-e10ab43b95f1-workload-certs" pod "b5499971-7479-4b2e-969c-e10ab43b95f1" (UID: "b5499971-7479-4b2e-969c-e10ab43b95f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:39:39.667497 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.667214 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6858a801-456a-4716-be05-8a3a7bd80334-kube-api-access-zwn7g" pod "6858a801-456a-4716-be05-8a3a7bd80334" (UID: "6858a801-456a-4716-be05-8a3a7bd80334"). InnerVolumeSpecName "kube-api-access-zwn7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:39:39.667497 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.667463 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-credential-socket" pod "0ef9c920-60ba-43ce-a4a2-2f5bc2731279" (UID: "0ef9c920-60ba-43ce-a4a2-2f5bc2731279"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:39:39.668046 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.667869 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-istiod-ca-cert" pod "0ef9c920-60ba-43ce-a4a2-2f5bc2731279" (UID: "0ef9c920-60ba-43ce-a4a2-2f5bc2731279"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:39:39.668412 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.668092 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-istio-envoy" pod "0ef9c920-60ba-43ce-a4a2-2f5bc2731279" (UID: "0ef9c920-60ba-43ce-a4a2-2f5bc2731279"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:39:39.668412 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.668248 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6858a801-456a-4716-be05-8a3a7bd80334-istio-data" pod "6858a801-456a-4716-be05-8a3a7bd80334" (UID: "6858a801-456a-4716-be05-8a3a7bd80334"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:39:39.668412 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.668386 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5499971-7479-4b2e-969c-e10ab43b95f1-istiod-ca-cert" pod "b5499971-7479-4b2e-969c-e10ab43b95f1" (UID: "b5499971-7479-4b2e-969c-e10ab43b95f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:39:39.668706 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.668681 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-istio-podinfo" pod "0ef9c920-60ba-43ce-a4a2-2f5bc2731279" (UID: "0ef9c920-60ba-43ce-a4a2-2f5bc2731279"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:39:39.668926 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.668889 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6858a801-456a-4716-be05-8a3a7bd80334-credential-socket" pod "6858a801-456a-4716-be05-8a3a7bd80334" (UID: "6858a801-456a-4716-be05-8a3a7bd80334"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:39:39.669841 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.669276 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d56da67-6cfc-49c4-802b-3c9579c87110-istio-data" pod "0d56da67-6cfc-49c4-802b-3c9579c87110" (UID: "0d56da67-6cfc-49c4-802b-3c9579c87110"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:39:39.669841 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.669470 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d56da67-6cfc-49c4-802b-3c9579c87110-workload-socket" pod "0d56da67-6cfc-49c4-802b-3c9579c87110" (UID: "0d56da67-6cfc-49c4-802b-3c9579c87110"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:39:39.669841 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.669568 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6858a801-456a-4716-be05-8a3a7bd80334-istio-envoy" pod "6858a801-456a-4716-be05-8a3a7bd80334" (UID: "6858a801-456a-4716-be05-8a3a7bd80334"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:39:39.671976 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.671945 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d56da67-6cfc-49c4-802b-3c9579c87110-istio-token" pod "0d56da67-6cfc-49c4-802b-3c9579c87110" (UID: "0d56da67-6cfc-49c4-802b-3c9579c87110"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:39:39.672078 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.672024 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d56da67-6cfc-49c4-802b-3c9579c87110-kube-api-access-zqdqw" pod "0d56da67-6cfc-49c4-802b-3c9579c87110" (UID: "0d56da67-6cfc-49c4-802b-3c9579c87110"). InnerVolumeSpecName "kube-api-access-zqdqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:39:39.672144 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.672098 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-istio-token" pod "0ef9c920-60ba-43ce-a4a2-2f5bc2731279" (UID: "0ef9c920-60ba-43ce-a4a2-2f5bc2731279"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:39:39.672418 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.672396 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-kube-api-access-qfv65" pod "0ef9c920-60ba-43ce-a4a2-2f5bc2731279" (UID: "0ef9c920-60ba-43ce-a4a2-2f5bc2731279"). InnerVolumeSpecName "kube-api-access-qfv65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:39:39.672651 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.672630 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5499971-7479-4b2e-969c-e10ab43b95f1-istio-token" pod "b5499971-7479-4b2e-969c-e10ab43b95f1" (UID: "b5499971-7479-4b2e-969c-e10ab43b95f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:39:39.673462 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.673434 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d56da67-6cfc-49c4-802b-3c9579c87110-istio-podinfo" pod "0d56da67-6cfc-49c4-802b-3c9579c87110" (UID: "0d56da67-6cfc-49c4-802b-3c9579c87110"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:39:39.673622 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.673504 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6858a801-456a-4716-be05-8a3a7bd80334-workload-certs" pod "6858a801-456a-4716-be05-8a3a7bd80334" (UID: "6858a801-456a-4716-be05-8a3a7bd80334"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:39:39.673677 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.673634 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d56da67-6cfc-49c4-802b-3c9579c87110-workload-certs" pod "0d56da67-6cfc-49c4-802b-3c9579c87110" (UID: "0d56da67-6cfc-49c4-802b-3c9579c87110"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:39:39.673714 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.673525 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6858a801-456a-4716-be05-8a3a7bd80334-istio-token" pod "6858a801-456a-4716-be05-8a3a7bd80334" (UID: "6858a801-456a-4716-be05-8a3a7bd80334"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:39:39.674310 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.674290 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5499971-7479-4b2e-969c-e10ab43b95f1-istio-podinfo" pod "b5499971-7479-4b2e-969c-e10ab43b95f1" (UID: "b5499971-7479-4b2e-969c-e10ab43b95f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:39:39.674881 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.674853 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d56da67-6cfc-49c4-802b-3c9579c87110-istio-envoy" pod "0d56da67-6cfc-49c4-802b-3c9579c87110" (UID: "0d56da67-6cfc-49c4-802b-3c9579c87110"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:39:39.678931 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.678903 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5499971-7479-4b2e-969c-e10ab43b95f1-istio-envoy" pod "b5499971-7479-4b2e-969c-e10ab43b95f1" (UID: "b5499971-7479-4b2e-969c-e10ab43b95f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:39:39.766957 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.766925 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:39:39.766957 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.766955 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:39:39.767211 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.766969 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d56da67-6cfc-49c4-802b-3c9579c87110-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:39:39.767211 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.766982 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d56da67-6cfc-49c4-802b-3c9579c87110-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:39:39.767211 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.766993 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d56da67-6cfc-49c4-802b-3c9579c87110-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:39:39.767211 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.767006 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:39:39.767211 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.767019 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d56da67-6cfc-49c4-802b-3c9579c87110-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:39:39.767211 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.767031 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6858a801-456a-4716-be05-8a3a7bd80334-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:39:39.767211 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.767044 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd7zh\" (UniqueName: \"kubernetes.io/projected/b5499971-7479-4b2e-969c-e10ab43b95f1-kube-api-access-qd7zh\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:39:39.767211 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.767055 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6858a801-456a-4716-be05-8a3a7bd80334-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:39:39.767211 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.767067 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5499971-7479-4b2e-969c-e10ab43b95f1-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:39:39.767211 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.767080 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwn7g\" (UniqueName: \"kubernetes.io/projected/6858a801-456a-4716-be05-8a3a7bd80334-kube-api-access-zwn7g\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:39:39.767211 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.767093 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5499971-7479-4b2e-969c-e10ab43b95f1-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:39:39.767211 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.767105 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6858a801-456a-4716-be05-8a3a7bd80334-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:39:39.767211 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.767116 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d56da67-6cfc-49c4-802b-3c9579c87110-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:39:39.767211 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.767128 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6858a801-456a-4716-be05-8a3a7bd80334-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:39:39.767211 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.767140 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:39:39.767211 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.767151 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6858a801-456a-4716-be05-8a3a7bd80334-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:39:39.767211 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.767162 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5499971-7479-4b2e-969c-e10ab43b95f1-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:39:39.767211 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.767174 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5499971-7479-4b2e-969c-e10ab43b95f1-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:39:39.767211 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.767188 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:39:39.767211 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.767201 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfv65\" (UniqueName: \"kubernetes.io/projected/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-kube-api-access-qfv65\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:39:39.767211 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.767213 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6858a801-456a-4716-be05-8a3a7bd80334-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:39:39.767932 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.767225 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d56da67-6cfc-49c4-802b-3c9579c87110-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:39:39.767932 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.767243 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6858a801-456a-4716-be05-8a3a7bd80334-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:39:39.767932 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.767256 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5499971-7479-4b2e-969c-e10ab43b95f1-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:39:39.767932 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.767270 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d56da67-6cfc-49c4-802b-3c9579c87110-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:39:39.767932 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.767283 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d56da67-6cfc-49c4-802b-3c9579c87110-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:39:39.767932 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.767297 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqdqw\" (UniqueName: \"kubernetes.io/projected/0d56da67-6cfc-49c4-802b-3c9579c87110-kube-api-access-zqdqw\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:39:39.767932 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.767310 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6858a801-456a-4716-be05-8a3a7bd80334-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:39:39.767932 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.767325 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5499971-7479-4b2e-969c-e10ab43b95f1-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:39:39.767932 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:39.767337 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ef9c920-60ba-43ce-a4a2-2f5bc2731279-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:39:40.028444 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:40.028350 6221 generic.go:356] "Generic (PLEG): container finished" podID="0ef9c920-60ba-43ce-a4a2-2f5bc2731279" containerID="25a46734f3ed22e9662fc8e7fdf7d563f4c419cccfa061acdc1347852a075f2b" exitCode=0 Oct 30 01:39:40.028444 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:40.028398 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb" Oct 30 01:39:40.028675 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:40.028435 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb" event={"ID":"0ef9c920-60ba-43ce-a4a2-2f5bc2731279","Type":"ContainerDied","Data":"25a46734f3ed22e9662fc8e7fdf7d563f4c419cccfa061acdc1347852a075f2b"} Oct 30 01:39:40.028675 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:40.028502 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb" event={"ID":"0ef9c920-60ba-43ce-a4a2-2f5bc2731279","Type":"ContainerDied","Data":"6f39d90e3d1fdba71a1d2d74287f8936be9230d4c78ce0cc0e2743f1c6dd0ded"} Oct 30 01:39:40.028675 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:40.028552 6221 scope.go:122] "RemoveContainer" containerID="25a46734f3ed22e9662fc8e7fdf7d563f4c419cccfa061acdc1347852a075f2b" Oct 30 01:39:40.029929 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:40.029906 6221 generic.go:356] "Generic (PLEG): container finished" podID="0d56da67-6cfc-49c4-802b-3c9579c87110" containerID="cdc620cdba9036461218c8eaf7939e0e3a185372e9ba03380aeac0cbfed1e20f" exitCode=0 Oct 30 01:39:40.030007 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:40.029951 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb" Oct 30 01:39:40.030461 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:40.030043 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb" event={"ID":"0d56da67-6cfc-49c4-802b-3c9579c87110","Type":"ContainerDied","Data":"cdc620cdba9036461218c8eaf7939e0e3a185372e9ba03380aeac0cbfed1e20f"} Oct 30 01:39:40.030631 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:40.030487 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb" event={"ID":"0d56da67-6cfc-49c4-802b-3c9579c87110","Type":"ContainerDied","Data":"ed63c0509ef13385eb9d4679a602ac9acaca1b3bf15f3dbe766e23f7bfdc0b8b"} Oct 30 01:39:40.031616 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:40.031592 6221 generic.go:356] "Generic (PLEG): container finished" podID="b5499971-7479-4b2e-969c-e10ab43b95f1" containerID="5f1ed4126b73545b5f734a0337a9383cfb6471dfff05ffda627d642e9db9de1d" exitCode=0 Oct 30 01:39:40.031747 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:40.031640 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6" Oct 30 01:39:40.031747 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:40.031667 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6" event={"ID":"b5499971-7479-4b2e-969c-e10ab43b95f1","Type":"ContainerDied","Data":"5f1ed4126b73545b5f734a0337a9383cfb6471dfff05ffda627d642e9db9de1d"} Oct 30 01:39:40.031747 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:40.031717 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6" event={"ID":"b5499971-7479-4b2e-969c-e10ab43b95f1","Type":"ContainerDied","Data":"5fb53eedd0cc1a8d77f2a4069b857ef0f29a3edae4b2421d8473d292270a7a09"} Oct 30 01:39:40.033363 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:40.033339 6221 generic.go:356] "Generic (PLEG): container finished" podID="6858a801-456a-4716-be05-8a3a7bd80334" containerID="3ab22f87b33c77d11f6f9ded0e2b60d140bdfb674713ad874369a98eaa5a0d72" exitCode=0 Oct 30 01:39:40.033471 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:40.033392 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq" Oct 30 01:39:40.033569 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:40.033494 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq" event={"ID":"6858a801-456a-4716-be05-8a3a7bd80334","Type":"ContainerDied","Data":"3ab22f87b33c77d11f6f9ded0e2b60d140bdfb674713ad874369a98eaa5a0d72"} Oct 30 01:39:40.033624 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:40.033573 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq" event={"ID":"6858a801-456a-4716-be05-8a3a7bd80334","Type":"ContainerDied","Data":"18da2a2704e18862fa7d1c18a5f1142485e647d6c6f18b959e90b5ee1daaa89a"} Oct 30 01:39:40.040137 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:40.040016 6221 scope.go:122] "RemoveContainer" containerID="25a46734f3ed22e9662fc8e7fdf7d563f4c419cccfa061acdc1347852a075f2b" Oct 30 01:39:40.040454 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:39:40.040429 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25a46734f3ed22e9662fc8e7fdf7d563f4c419cccfa061acdc1347852a075f2b\": container with ID starting with 25a46734f3ed22e9662fc8e7fdf7d563f4c419cccfa061acdc1347852a075f2b not found: ID does not exist" containerID="25a46734f3ed22e9662fc8e7fdf7d563f4c419cccfa061acdc1347852a075f2b" Oct 30 01:39:40.040593 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:40.040463 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25a46734f3ed22e9662fc8e7fdf7d563f4c419cccfa061acdc1347852a075f2b"} err="failed to get container status \"25a46734f3ed22e9662fc8e7fdf7d563f4c419cccfa061acdc1347852a075f2b\": rpc error: code = NotFound desc = could not find container \"25a46734f3ed22e9662fc8e7fdf7d563f4c419cccfa061acdc1347852a075f2b\": container with ID starting with 25a46734f3ed22e9662fc8e7fdf7d563f4c419cccfa061acdc1347852a075f2b not found: ID does not exist" Oct 30 01:39:40.040593 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:40.040486 6221 scope.go:122] "RemoveContainer" containerID="cdc620cdba9036461218c8eaf7939e0e3a185372e9ba03380aeac0cbfed1e20f" Oct 30 01:39:40.050540 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:40.050513 6221 scope.go:122] "RemoveContainer" containerID="cdc620cdba9036461218c8eaf7939e0e3a185372e9ba03380aeac0cbfed1e20f" Oct 30 01:39:40.050823 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:39:40.050803 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdc620cdba9036461218c8eaf7939e0e3a185372e9ba03380aeac0cbfed1e20f\": container with ID starting with cdc620cdba9036461218c8eaf7939e0e3a185372e9ba03380aeac0cbfed1e20f not found: ID does not exist" containerID="cdc620cdba9036461218c8eaf7939e0e3a185372e9ba03380aeac0cbfed1e20f" Oct 30 01:39:40.050876 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:40.050835 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdc620cdba9036461218c8eaf7939e0e3a185372e9ba03380aeac0cbfed1e20f"} err="failed to get container status \"cdc620cdba9036461218c8eaf7939e0e3a185372e9ba03380aeac0cbfed1e20f\": rpc error: code = NotFound desc = could not find container \"cdc620cdba9036461218c8eaf7939e0e3a185372e9ba03380aeac0cbfed1e20f\": container with ID starting with cdc620cdba9036461218c8eaf7939e0e3a185372e9ba03380aeac0cbfed1e20f not found: ID does not exist" Oct 30 01:39:40.050876 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:40.050856 6221 scope.go:122] "RemoveContainer" containerID="5f1ed4126b73545b5f734a0337a9383cfb6471dfff05ffda627d642e9db9de1d" Oct 30 01:39:40.059713 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:40.059696 6221 scope.go:122] "RemoveContainer" containerID="5f1ed4126b73545b5f734a0337a9383cfb6471dfff05ffda627d642e9db9de1d" Oct 30 01:39:40.059990 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:39:40.059966 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f1ed4126b73545b5f734a0337a9383cfb6471dfff05ffda627d642e9db9de1d\": container with ID starting with 5f1ed4126b73545b5f734a0337a9383cfb6471dfff05ffda627d642e9db9de1d not found: ID does not exist" containerID="5f1ed4126b73545b5f734a0337a9383cfb6471dfff05ffda627d642e9db9de1d" Oct 30 01:39:40.060082 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:40.059996 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f1ed4126b73545b5f734a0337a9383cfb6471dfff05ffda627d642e9db9de1d"} err="failed to get container status \"5f1ed4126b73545b5f734a0337a9383cfb6471dfff05ffda627d642e9db9de1d\": rpc error: code = NotFound desc = could not find container \"5f1ed4126b73545b5f734a0337a9383cfb6471dfff05ffda627d642e9db9de1d\": container with ID starting with 5f1ed4126b73545b5f734a0337a9383cfb6471dfff05ffda627d642e9db9de1d not found: ID does not exist" Oct 30 01:39:40.060082 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:40.060013 6221 scope.go:122] "RemoveContainer" containerID="3ab22f87b33c77d11f6f9ded0e2b60d140bdfb674713ad874369a98eaa5a0d72" Oct 30 01:39:40.069469 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:40.069446 6221 scope.go:122] "RemoveContainer" containerID="3ab22f87b33c77d11f6f9ded0e2b60d140bdfb674713ad874369a98eaa5a0d72" Oct 30 01:39:40.069812 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:39:40.069783 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ab22f87b33c77d11f6f9ded0e2b60d140bdfb674713ad874369a98eaa5a0d72\": container with ID starting with 3ab22f87b33c77d11f6f9ded0e2b60d140bdfb674713ad874369a98eaa5a0d72 not found: ID does not exist" containerID="3ab22f87b33c77d11f6f9ded0e2b60d140bdfb674713ad874369a98eaa5a0d72" Oct 30 01:39:40.069894 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:40.069812 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ab22f87b33c77d11f6f9ded0e2b60d140bdfb674713ad874369a98eaa5a0d72"} err="failed to get container status \"3ab22f87b33c77d11f6f9ded0e2b60d140bdfb674713ad874369a98eaa5a0d72\": rpc error: code = NotFound desc = could not find container \"3ab22f87b33c77d11f6f9ded0e2b60d140bdfb674713ad874369a98eaa5a0d72\": container with ID starting with 3ab22f87b33c77d11f6f9ded0e2b60d140bdfb674713ad874369a98eaa5a0d72 not found: ID does not exist" Oct 30 01:39:40.116188 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:40.116162 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb"] Oct 30 01:39:40.128238 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:40.128214 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-dm9bb"] Oct 30 01:39:40.157054 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:40.157026 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb"] Oct 30 01:39:40.159569 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:40.159547 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-2gzmb"] Oct 30 01:39:40.192212 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:40.192180 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq"] Oct 30 01:39:40.197358 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:40.197331 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-5dcmq"] Oct 30 01:39:40.235369 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:40.235336 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6"] Oct 30 01:39:40.238158 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:40.238134 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-z85f6"] Oct 30 01:39:40.547334 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:40.547301 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d56da67-6cfc-49c4-802b-3c9579c87110" path="/var/lib/kubelet/pods/0d56da67-6cfc-49c4-802b-3c9579c87110/volumes" Oct 30 01:39:40.547829 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:40.547813 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ef9c920-60ba-43ce-a4a2-2f5bc2731279" path="/var/lib/kubelet/pods/0ef9c920-60ba-43ce-a4a2-2f5bc2731279/volumes" Oct 30 01:39:40.548246 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:40.548234 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6858a801-456a-4716-be05-8a3a7bd80334" path="/var/lib/kubelet/pods/6858a801-456a-4716-be05-8a3a7bd80334/volumes" Oct 30 01:39:40.548988 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:40.548972 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5499971-7479-4b2e-969c-e10ab43b95f1" path="/var/lib/kubelet/pods/b5499971-7479-4b2e-969c-e10ab43b95f1/volumes" Oct 30 01:39:41.723412 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:41.723374 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f01cbc-76b448f6-jkd4f"] Oct 30 01:39:41.724277 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:41.724252 6221 cpu_manager.go:414] "containerMap: removing container" podUID="0ef9c920-60ba-43ce-a4a2-2f5bc2731279" containerName="istio-proxy" Oct 30 01:39:41.724277 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:41.724277 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ef9c920-60ba-43ce-a4a2-2f5bc2731279" containerName="istio-proxy" Oct 30 01:39:41.724421 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:41.724294 6221 cpu_manager.go:414] "containerMap: removing container" podUID="0d56da67-6cfc-49c4-802b-3c9579c87110" containerName="istio-proxy" Oct 30 01:39:41.724421 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:41.724300 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d56da67-6cfc-49c4-802b-3c9579c87110" containerName="istio-proxy" Oct 30 01:39:41.724421 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:41.724313 6221 cpu_manager.go:414] "containerMap: removing container" podUID="b5499971-7479-4b2e-969c-e10ab43b95f1" containerName="istio-proxy" Oct 30 01:39:41.724421 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:41.724319 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5499971-7479-4b2e-969c-e10ab43b95f1" containerName="istio-proxy" Oct 30 01:39:41.724421 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:41.724336 6221 cpu_manager.go:414] "containerMap: removing container" podUID="6858a801-456a-4716-be05-8a3a7bd80334" containerName="istio-proxy" Oct 30 01:39:41.724421 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:41.724341 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6858a801-456a-4716-be05-8a3a7bd80334" containerName="istio-proxy" Oct 30 01:39:41.724632 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:41.724451 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d56da67-6cfc-49c4-802b-3c9579c87110" containerName="istio-proxy" Oct 30 01:39:41.724632 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:41.724461 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5499971-7479-4b2e-969c-e10ab43b95f1" containerName="istio-proxy" Oct 30 01:39:41.724632 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:41.724469 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="6858a801-456a-4716-be05-8a3a7bd80334" containerName="istio-proxy" Oct 30 01:39:41.724632 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:41.724482 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ef9c920-60ba-43ce-a4a2-2f5bc2731279" containerName="istio-proxy" Oct 30 01:39:41.730118 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:41.730094 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f01cbc-76b448f6-jkd4f" Oct 30 01:39:41.747327 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:41.747300 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f01cbc-76b448f6-jkd4f"] Oct 30 01:39:41.761312 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:41.757132 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f01cbc\"" Oct 30 01:39:41.782246 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:41.782215 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f01cbc-664df47bc-pgtv2"] Oct 30 01:39:41.786558 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:41.786522 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f01cbc-664df47bc-pgtv2" Oct 30 01:39:41.795429 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:41.795404 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0a4deee-d48d-4533-a043-34d7ca72705f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f01cbc-76b448f6-jkd4f\" (UID: \"d0a4deee-d48d-4533-a043-34d7ca72705f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f01cbc-76b448f6-jkd4f" Oct 30 01:39:41.795641 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:41.795625 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9msd\" (UniqueName: \"kubernetes.io/projected/d0a4deee-d48d-4533-a043-34d7ca72705f-kube-api-access-n9msd\") pod \"payload-processing-e2e-ipp-b-f01cbc-76b448f6-jkd4f\" (UID: \"d0a4deee-d48d-4533-a043-34d7ca72705f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f01cbc-76b448f6-jkd4f" Oct 30 01:39:41.807717 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:41.807692 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f01cbc-664df47bc-pgtv2"] Oct 30 01:39:41.905560 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:41.900514 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7c2b8d5-5ea1-4ddd-8ad9-fd98031a9f8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f01cbc-664df47bc-pgtv2\" (UID: \"c7c2b8d5-5ea1-4ddd-8ad9-fd98031a9f8c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f01cbc-664df47bc-pgtv2" Oct 30 01:39:41.926558 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:41.909649 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcpb4\" (UniqueName: \"kubernetes.io/projected/c7c2b8d5-5ea1-4ddd-8ad9-fd98031a9f8c-kube-api-access-hcpb4\") pod \"payload-pre-processing-e2e-ipp-b-f01cbc-664df47bc-pgtv2\" (UID: \"c7c2b8d5-5ea1-4ddd-8ad9-fd98031a9f8c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f01cbc-664df47bc-pgtv2" Oct 30 01:39:41.927055 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:41.927014 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9msd\" (UniqueName: \"kubernetes.io/projected/d0a4deee-d48d-4533-a043-34d7ca72705f-kube-api-access-n9msd\") pod \"payload-processing-e2e-ipp-b-f01cbc-76b448f6-jkd4f\" (UID: \"d0a4deee-d48d-4533-a043-34d7ca72705f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f01cbc-76b448f6-jkd4f" Oct 30 01:39:41.951554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:41.927339 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0a4deee-d48d-4533-a043-34d7ca72705f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f01cbc-76b448f6-jkd4f\" (UID: \"d0a4deee-d48d-4533-a043-34d7ca72705f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f01cbc-76b448f6-jkd4f" Oct 30 01:39:41.951915 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:41.910886 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f01cbc-dockercfg-xjgml\"" Oct 30 01:39:41.952024 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:41.949450 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0a4deee-d48d-4533-a043-34d7ca72705f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f01cbc-76b448f6-jkd4f\" (UID: \"d0a4deee-d48d-4533-a043-34d7ca72705f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f01cbc-76b448f6-jkd4f" Oct 30 01:39:41.976395 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:41.976298 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9msd\" (UniqueName: \"kubernetes.io/projected/d0a4deee-d48d-4533-a043-34d7ca72705f-kube-api-access-n9msd\") pod \"payload-processing-e2e-ipp-b-f01cbc-76b448f6-jkd4f\" (UID: \"d0a4deee-d48d-4533-a043-34d7ca72705f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f01cbc-76b448f6-jkd4f" Oct 30 01:39:42.052162 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:42.051891 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f01cbc-76b448f6-jkd4f" Oct 30 01:39:42.055810 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:42.055631 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7c2b8d5-5ea1-4ddd-8ad9-fd98031a9f8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f01cbc-664df47bc-pgtv2\" (UID: \"c7c2b8d5-5ea1-4ddd-8ad9-fd98031a9f8c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f01cbc-664df47bc-pgtv2" Oct 30 01:39:42.056563 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:42.056492 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7c2b8d5-5ea1-4ddd-8ad9-fd98031a9f8c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f01cbc-664df47bc-pgtv2\" (UID: \"c7c2b8d5-5ea1-4ddd-8ad9-fd98031a9f8c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f01cbc-664df47bc-pgtv2" Oct 30 01:39:42.073850 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:42.066190 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcpb4\" (UniqueName: \"kubernetes.io/projected/c7c2b8d5-5ea1-4ddd-8ad9-fd98031a9f8c-kube-api-access-hcpb4\") pod \"payload-pre-processing-e2e-ipp-b-f01cbc-664df47bc-pgtv2\" (UID: \"c7c2b8d5-5ea1-4ddd-8ad9-fd98031a9f8c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f01cbc-664df47bc-pgtv2" Oct 30 01:39:42.090344 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:42.090295 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcpb4\" (UniqueName: \"kubernetes.io/projected/c7c2b8d5-5ea1-4ddd-8ad9-fd98031a9f8c-kube-api-access-hcpb4\") pod \"payload-pre-processing-e2e-ipp-b-f01cbc-664df47bc-pgtv2\" (UID: \"c7c2b8d5-5ea1-4ddd-8ad9-fd98031a9f8c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f01cbc-664df47bc-pgtv2" Oct 30 01:39:42.101494 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:42.101460 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f01cbc-664df47bc-pgtv2" Oct 30 01:39:42.388132 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:42.388104 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f01cbc-76b448f6-jkd4f"] Oct 30 01:39:42.390551 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:39:42.390508 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0a4deee_d48d_4533_a043_34d7ca72705f.slice/crio-e77e53bf19378cd463d140fe50081ab9a45f2ef401211d10507dfcc3bc653d29 WatchSource:0}: Error finding container e77e53bf19378cd463d140fe50081ab9a45f2ef401211d10507dfcc3bc653d29: Status 404 returned error can't find the container with id e77e53bf19378cd463d140fe50081ab9a45f2ef401211d10507dfcc3bc653d29 Oct 30 01:39:42.416705 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:42.416681 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f01cbc-664df47bc-pgtv2"] Oct 30 01:39:42.421743 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:39:42.421716 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7c2b8d5_5ea1_4ddd_8ad9_fd98031a9f8c.slice/crio-dfd68e29a358ee0093be9e6b04d02857db235a91cffccf085523bfe32bd9352a WatchSource:0}: Error finding container dfd68e29a358ee0093be9e6b04d02857db235a91cffccf085523bfe32bd9352a: Status 404 returned error can't find the container with id dfd68e29a358ee0093be9e6b04d02857db235a91cffccf085523bfe32bd9352a Oct 30 01:39:42.763462 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:42.763382 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-18560b-698849667d-6tdwg"] Oct 30 01:39:42.764794 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:42.763637 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18560b-698849667d-6tdwg" podUID="cf83d1bd-9ffa-4367-b8ce-80978585a9f2" containerName="maas-api" containerID="cri-o://22a9f76deeeb7156138f0226b20bcfe3098cf5242dd0e3ea2935eba70805d0b2" gracePeriod=30 Oct 30 01:39:43.106625 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.097393 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18560b-698849667d-6tdwg" Oct 30 01:39:43.143215 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.142719 6221 generic.go:356] "Generic (PLEG): container finished" podID="cf83d1bd-9ffa-4367-b8ce-80978585a9f2" containerID="22a9f76deeeb7156138f0226b20bcfe3098cf5242dd0e3ea2935eba70805d0b2" exitCode=0 Oct 30 01:39:43.143215 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.142913 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18560b-698849667d-6tdwg" Oct 30 01:39:43.143215 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.143039 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18560b-698849667d-6tdwg" event={"ID":"cf83d1bd-9ffa-4367-b8ce-80978585a9f2","Type":"ContainerDied","Data":"22a9f76deeeb7156138f0226b20bcfe3098cf5242dd0e3ea2935eba70805d0b2"} Oct 30 01:39:43.143215 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.143079 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-18560b-698849667d-6tdwg" event={"ID":"cf83d1bd-9ffa-4367-b8ce-80978585a9f2","Type":"ContainerDied","Data":"92250ca3386f379fa02474dca24d735d9a98a5c5027755a9bcbf657d1429926c"} Oct 30 01:39:43.143215 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.143105 6221 scope.go:122] "RemoveContainer" containerID="22a9f76deeeb7156138f0226b20bcfe3098cf5242dd0e3ea2935eba70805d0b2" Oct 30 01:39:43.147332 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.146459 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f01cbc-76b448f6-jkd4f" event={"ID":"d0a4deee-d48d-4533-a043-34d7ca72705f","Type":"ContainerStarted","Data":"3c6eef4a00d9ddcd64f8953e5f2fedc6b41420759864504e8389b3ea13496c61"} Oct 30 01:39:43.147332 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.146510 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f01cbc-76b448f6-jkd4f" event={"ID":"d0a4deee-d48d-4533-a043-34d7ca72705f","Type":"ContainerStarted","Data":"e77e53bf19378cd463d140fe50081ab9a45f2ef401211d10507dfcc3bc653d29"} Oct 30 01:39:43.147332 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.146633 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f01cbc-76b448f6-jkd4f" Oct 30 01:39:43.148367 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.148275 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f01cbc-664df47bc-pgtv2" event={"ID":"c7c2b8d5-5ea1-4ddd-8ad9-fd98031a9f8c","Type":"ContainerStarted","Data":"78b6e9e4d99b38c3a133faf49dcc271e79d362385be46a2d5521e2036cf448fd"} Oct 30 01:39:43.148367 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.148320 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f01cbc-664df47bc-pgtv2" event={"ID":"c7c2b8d5-5ea1-4ddd-8ad9-fd98031a9f8c","Type":"ContainerStarted","Data":"dfd68e29a358ee0093be9e6b04d02857db235a91cffccf085523bfe32bd9352a"} Oct 30 01:39:43.148638 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.148583 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f01cbc-664df47bc-pgtv2" Oct 30 01:39:43.175566 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.164358 6221 scope.go:122] "RemoveContainer" containerID="22a9f76deeeb7156138f0226b20bcfe3098cf5242dd0e3ea2935eba70805d0b2" Oct 30 01:39:43.175566 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:39:43.165515 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22a9f76deeeb7156138f0226b20bcfe3098cf5242dd0e3ea2935eba70805d0b2\": container with ID starting with 22a9f76deeeb7156138f0226b20bcfe3098cf5242dd0e3ea2935eba70805d0b2 not found: ID does not exist" containerID="22a9f76deeeb7156138f0226b20bcfe3098cf5242dd0e3ea2935eba70805d0b2" Oct 30 01:39:43.175566 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.165569 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22a9f76deeeb7156138f0226b20bcfe3098cf5242dd0e3ea2935eba70805d0b2"} err="failed to get container status \"22a9f76deeeb7156138f0226b20bcfe3098cf5242dd0e3ea2935eba70805d0b2\": rpc error: code = NotFound desc = could not find container \"22a9f76deeeb7156138f0226b20bcfe3098cf5242dd0e3ea2935eba70805d0b2\": container with ID starting with 22a9f76deeeb7156138f0226b20bcfe3098cf5242dd0e3ea2935eba70805d0b2 not found: ID does not exist" Oct 30 01:39:43.277593 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.248872 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf83d1bd-9ffa-4367-b8ce-80978585a9f2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf83d1bd-9ffa-4367-b8ce-80978585a9f2-maas-api-tls\") pod \"cf83d1bd-9ffa-4367-b8ce-80978585a9f2\" (UID: \"cf83d1bd-9ffa-4367-b8ce-80978585a9f2\") " Oct 30 01:39:43.277593 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.249041 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf83d1bd-9ffa-4367-b8ce-80978585a9f2-kube-api-access-kqpxx\" (UniqueName: \"kubernetes.io/projected/cf83d1bd-9ffa-4367-b8ce-80978585a9f2-kube-api-access-kqpxx\") pod \"cf83d1bd-9ffa-4367-b8ce-80978585a9f2\" (UID: \"cf83d1bd-9ffa-4367-b8ce-80978585a9f2\") " Oct 30 01:39:43.277593 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.249082 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf83d1bd-9ffa-4367-b8ce-80978585a9f2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf83d1bd-9ffa-4367-b8ce-80978585a9f2-openshift-service-ca\") pod \"cf83d1bd-9ffa-4367-b8ce-80978585a9f2\" (UID: \"cf83d1bd-9ffa-4367-b8ce-80978585a9f2\") " Oct 30 01:39:43.277593 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.249151 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf83d1bd-9ffa-4367-b8ce-80978585a9f2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf83d1bd-9ffa-4367-b8ce-80978585a9f2-metrics-certs\") pod \"cf83d1bd-9ffa-4367-b8ce-80978585a9f2\" (UID: \"cf83d1bd-9ffa-4367-b8ce-80978585a9f2\") " Oct 30 01:39:43.322965 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.320287 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf83d1bd-9ffa-4367-b8ce-80978585a9f2-metrics-certs" pod "cf83d1bd-9ffa-4367-b8ce-80978585a9f2" (UID: "cf83d1bd-9ffa-4367-b8ce-80978585a9f2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:39:43.322965 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.320691 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf83d1bd-9ffa-4367-b8ce-80978585a9f2-openshift-service-ca" pod "cf83d1bd-9ffa-4367-b8ce-80978585a9f2" (UID: "cf83d1bd-9ffa-4367-b8ce-80978585a9f2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:39:43.362222 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.361868 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf83d1bd-9ffa-4367-b8ce-80978585a9f2-maas-api-tls" pod "cf83d1bd-9ffa-4367-b8ce-80978585a9f2" (UID: "cf83d1bd-9ffa-4367-b8ce-80978585a9f2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:39:43.363162 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.363092 6221 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf83d1bd-9ffa-4367-b8ce-80978585a9f2-openshift-service-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:39:43.363162 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.363120 6221 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf83d1bd-9ffa-4367-b8ce-80978585a9f2-metrics-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:39:43.363162 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.363136 6221 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf83d1bd-9ffa-4367-b8ce-80978585a9f2-maas-api-tls\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:39:43.366254 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.366212 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf83d1bd-9ffa-4367-b8ce-80978585a9f2-kube-api-access-kqpxx" pod "cf83d1bd-9ffa-4367-b8ce-80978585a9f2" (UID: "cf83d1bd-9ffa-4367-b8ce-80978585a9f2"). InnerVolumeSpecName "kube-api-access-kqpxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:39:43.367734 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.367684 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f01cbc-664df47bc-pgtv2" podStartSLOduration=2.367667473 podStartE2EDuration="2.367667473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:39:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:39:43.206015706 +0000 UTC m=+1306.784303044" watchObservedRunningTime="2026-10-30 01:39:43.367667473 +0000 UTC m=+1306.945954811" Oct 30 01:39:43.375565 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.369031 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f01cbc-76b448f6-jkd4f" podStartSLOduration=2.369018993 podStartE2EDuration="2.369018993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:39:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:39:43.295318497 +0000 UTC m=+1306.873605833" watchObservedRunningTime="2026-10-30 01:39:43.369018993 +0000 UTC m=+1306.947306330" Oct 30 01:39:43.491703 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.484724 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqpxx\" (UniqueName: \"kubernetes.io/projected/cf83d1bd-9ffa-4367-b8ce-80978585a9f2-kube-api-access-kqpxx\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:39:43.542109 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.542072 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db534-5b994486c7-lb44h"] Oct 30 01:39:43.542738 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.542719 6221 cpu_manager.go:414] "containerMap: removing container" podUID="cf83d1bd-9ffa-4367-b8ce-80978585a9f2" containerName="maas-api" Oct 30 01:39:43.542832 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.542740 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf83d1bd-9ffa-4367-b8ce-80978585a9f2" containerName="maas-api" Oct 30 01:39:43.542916 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.542900 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf83d1bd-9ffa-4367-b8ce-80978585a9f2" containerName="maas-api" Oct 30 01:39:43.547464 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.547443 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db534-5b994486c7-lb44h" Oct 30 01:39:43.548198 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.548179 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-18560b-698849667d-6tdwg"] Oct 30 01:39:43.554684 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.554655 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-18560b-698849667d-6tdwg"] Oct 30 01:39:43.562099 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.562076 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-0db534-serving-cert\"" Oct 30 01:39:43.564206 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.564179 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db534-5b994486c7-lb44h"] Oct 30 01:39:43.688195 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.688086 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d-metrics-certs\") pod \"maas-api-e2e-ipp-b-0db534-5b994486c7-lb44h\" (UID: \"ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db534-5b994486c7-lb44h" Oct 30 01:39:43.688381 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.688233 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0db534-5b994486c7-lb44h\" (UID: \"ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db534-5b994486c7-lb44h" Oct 30 01:39:43.688381 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.688360 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0db534-5b994486c7-lb44h\" (UID: \"ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db534-5b994486c7-lb44h" Oct 30 01:39:43.688510 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.688398 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8dth\" (UniqueName: \"kubernetes.io/projected/ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d-kube-api-access-d8dth\") pod \"maas-api-e2e-ipp-b-0db534-5b994486c7-lb44h\" (UID: \"ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db534-5b994486c7-lb44h" Oct 30 01:39:43.793808 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.793774 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0db534-5b994486c7-lb44h\" (UID: \"ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db534-5b994486c7-lb44h" Oct 30 01:39:43.795744 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.795695 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0db534-5b994486c7-lb44h\" (UID: \"ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db534-5b994486c7-lb44h" Oct 30 01:39:43.804208 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.804179 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8dth\" (UniqueName: \"kubernetes.io/projected/ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d-kube-api-access-d8dth\") pod \"maas-api-e2e-ipp-b-0db534-5b994486c7-lb44h\" (UID: \"ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db534-5b994486c7-lb44h" Oct 30 01:39:43.804581 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.795500 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0db534-5b994486c7-lb44h\" (UID: \"ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db534-5b994486c7-lb44h" Oct 30 01:39:43.805146 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.804920 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d-metrics-certs\") pod \"maas-api-e2e-ipp-b-0db534-5b994486c7-lb44h\" (UID: \"ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db534-5b994486c7-lb44h" Oct 30 01:39:43.806210 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.806186 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0db534-5b994486c7-lb44h\" (UID: \"ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db534-5b994486c7-lb44h" Oct 30 01:39:43.833382 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.833345 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8dth\" (UniqueName: \"kubernetes.io/projected/ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d-kube-api-access-d8dth\") pod \"maas-api-e2e-ipp-b-0db534-5b994486c7-lb44h\" (UID: \"ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db534-5b994486c7-lb44h" Oct 30 01:39:43.856163 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.856129 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d-metrics-certs\") pod \"maas-api-e2e-ipp-b-0db534-5b994486c7-lb44h\" (UID: \"ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db534-5b994486c7-lb44h" Oct 30 01:39:43.867566 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:43.866791 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db534-5b994486c7-lb44h" Oct 30 01:39:44.169037 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:44.169004 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db534-5b994486c7-lb44h"] Oct 30 01:39:44.172262 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:39:44.172237 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab33e5bd_3b6d_46d9_ab25_3c4a8a03a88d.slice/crio-4ddd9b8eead0979a5d84d91ee6c6fdc39555c455e23d74d3b445d949ed04b5f7 WatchSource:0}: Error finding container 4ddd9b8eead0979a5d84d91ee6c6fdc39555c455e23d74d3b445d949ed04b5f7: Status 404 returned error can't find the container with id 4ddd9b8eead0979a5d84d91ee6c6fdc39555c455e23d74d3b445d949ed04b5f7 Oct 30 01:39:44.548970 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:44.548942 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf83d1bd-9ffa-4367-b8ce-80978585a9f2" path="/var/lib/kubelet/pods/cf83d1bd-9ffa-4367-b8ce-80978585a9f2/volumes" Oct 30 01:39:45.162738 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:45.162695 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db534-5b994486c7-lb44h" event={"ID":"ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d","Type":"ContainerStarted","Data":"4fdb5a5741bedf42d486abf36e40cbab16970c3d5282559ff1857a48e6b043fc"} Oct 30 01:39:45.163226 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:45.162870 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db534-5b994486c7-lb44h" Oct 30 01:39:45.163226 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:45.162904 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db534-5b994486c7-lb44h" event={"ID":"ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d","Type":"ContainerStarted","Data":"4ddd9b8eead0979a5d84d91ee6c6fdc39555c455e23d74d3b445d949ed04b5f7"} Oct 30 01:39:45.203895 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:45.203836 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db534-5b994486c7-lb44h" podStartSLOduration=1.833415106 podStartE2EDuration="2.203820119s" totalImagesPullingTime="370.405013ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:39:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:39:45.202115619 +0000 UTC m=+1308.780402965" watchObservedRunningTime="2026-10-30 01:39:45.203820119 +0000 UTC m=+1308.782107455" Oct 30 01:39:48.136206 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:48.135136 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:39:51.174494 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:51.174465 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db534-5b994486c7-lb44h" Oct 30 01:39:51.566131 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:39:51.566093 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:39:54.158522 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:54.158490 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f01cbc-664df47bc-pgtv2" Oct 30 01:39:54.158932 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:39:54.158715 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f01cbc-76b448f6-jkd4f" Oct 30 01:40:05.956488 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:05.956452 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg"] Oct 30 01:40:05.962435 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:05.962406 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" Oct 30 01:40:05.983718 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:05.983686 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-2ac18f-openshift-default-dockercfg-6mhj6\"" Oct 30 01:40:05.999646 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:05.999613 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg"] Oct 30 01:40:06.078453 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:06.078411 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cda4f86d-d23b-4da5-824e-cacea3b08ab7-workload-certs\") pod \"e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg\" (UID: \"cda4f86d-d23b-4da5-824e-cacea3b08ab7\") " pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" Oct 30 01:40:06.078675 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:06.078470 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cda4f86d-d23b-4da5-824e-cacea3b08ab7-istiod-ca-cert\") pod \"e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg\" (UID: \"cda4f86d-d23b-4da5-824e-cacea3b08ab7\") " pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" Oct 30 01:40:06.078675 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:06.078616 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cda4f86d-d23b-4da5-824e-cacea3b08ab7-workload-socket\") pod \"e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg\" (UID: \"cda4f86d-d23b-4da5-824e-cacea3b08ab7\") " pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" Oct 30 01:40:06.078775 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:06.078721 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwgns\" (UniqueName: \"kubernetes.io/projected/cda4f86d-d23b-4da5-824e-cacea3b08ab7-kube-api-access-lwgns\") pod \"e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg\" (UID: \"cda4f86d-d23b-4da5-824e-cacea3b08ab7\") " pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" Oct 30 01:40:06.078844 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:06.078825 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cda4f86d-d23b-4da5-824e-cacea3b08ab7-credential-socket\") pod \"e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg\" (UID: \"cda4f86d-d23b-4da5-824e-cacea3b08ab7\") " pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" Oct 30 01:40:06.078895 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:06.078862 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cda4f86d-d23b-4da5-824e-cacea3b08ab7-istio-podinfo\") pod \"e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg\" (UID: \"cda4f86d-d23b-4da5-824e-cacea3b08ab7\") " pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" Oct 30 01:40:06.078930 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:06.078890 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cda4f86d-d23b-4da5-824e-cacea3b08ab7-istio-envoy\") pod \"e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg\" (UID: \"cda4f86d-d23b-4da5-824e-cacea3b08ab7\") " pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" Oct 30 01:40:06.079026 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:06.079011 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cda4f86d-d23b-4da5-824e-cacea3b08ab7-istio-token\") pod \"e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg\" (UID: \"cda4f86d-d23b-4da5-824e-cacea3b08ab7\") " pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" Oct 30 01:40:06.079069 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:06.079045 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cda4f86d-d23b-4da5-824e-cacea3b08ab7-istio-data\") pod \"e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg\" (UID: \"cda4f86d-d23b-4da5-824e-cacea3b08ab7\") " pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" Oct 30 01:40:06.189554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:06.188083 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cda4f86d-d23b-4da5-824e-cacea3b08ab7-istio-podinfo\") pod \"e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg\" (UID: \"cda4f86d-d23b-4da5-824e-cacea3b08ab7\") " pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" Oct 30 01:40:06.189554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:06.188135 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cda4f86d-d23b-4da5-824e-cacea3b08ab7-istio-envoy\") pod \"e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg\" (UID: \"cda4f86d-d23b-4da5-824e-cacea3b08ab7\") " pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" Oct 30 01:40:06.189554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:06.188264 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cda4f86d-d23b-4da5-824e-cacea3b08ab7-istio-token\") pod \"e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg\" (UID: \"cda4f86d-d23b-4da5-824e-cacea3b08ab7\") " pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" Oct 30 01:40:06.189554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:06.188296 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cda4f86d-d23b-4da5-824e-cacea3b08ab7-istio-data\") pod \"e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg\" (UID: \"cda4f86d-d23b-4da5-824e-cacea3b08ab7\") " pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" Oct 30 01:40:06.189554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:06.188364 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cda4f86d-d23b-4da5-824e-cacea3b08ab7-workload-certs\") pod \"e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg\" (UID: \"cda4f86d-d23b-4da5-824e-cacea3b08ab7\") " pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" Oct 30 01:40:06.189554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:06.188395 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cda4f86d-d23b-4da5-824e-cacea3b08ab7-istiod-ca-cert\") pod \"e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg\" (UID: \"cda4f86d-d23b-4da5-824e-cacea3b08ab7\") " pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" Oct 30 01:40:06.189554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:06.188430 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cda4f86d-d23b-4da5-824e-cacea3b08ab7-workload-socket\") pod \"e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg\" (UID: \"cda4f86d-d23b-4da5-824e-cacea3b08ab7\") " pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" Oct 30 01:40:06.189554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:06.188500 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwgns\" (UniqueName: \"kubernetes.io/projected/cda4f86d-d23b-4da5-824e-cacea3b08ab7-kube-api-access-lwgns\") pod \"e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg\" (UID: \"cda4f86d-d23b-4da5-824e-cacea3b08ab7\") " pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" Oct 30 01:40:06.201561 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:06.196382 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cda4f86d-d23b-4da5-824e-cacea3b08ab7-workload-certs\") pod \"e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg\" (UID: \"cda4f86d-d23b-4da5-824e-cacea3b08ab7\") " pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" Oct 30 01:40:06.201561 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:06.196696 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cda4f86d-d23b-4da5-824e-cacea3b08ab7-istio-data\") pod \"e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg\" (UID: \"cda4f86d-d23b-4da5-824e-cacea3b08ab7\") " pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" Oct 30 01:40:06.201561 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:06.197408 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cda4f86d-d23b-4da5-824e-cacea3b08ab7-istiod-ca-cert\") pod \"e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg\" (UID: \"cda4f86d-d23b-4da5-824e-cacea3b08ab7\") " pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" Oct 30 01:40:06.215610 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:06.210745 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cda4f86d-d23b-4da5-824e-cacea3b08ab7-credential-socket\") pod \"e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg\" (UID: \"cda4f86d-d23b-4da5-824e-cacea3b08ab7\") " pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" Oct 30 01:40:06.215610 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:06.213274 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cda4f86d-d23b-4da5-824e-cacea3b08ab7-workload-socket\") pod \"e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg\" (UID: \"cda4f86d-d23b-4da5-824e-cacea3b08ab7\") " pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" Oct 30 01:40:06.215610 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:06.213568 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cda4f86d-d23b-4da5-824e-cacea3b08ab7-credential-socket\") pod \"e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg\" (UID: \"cda4f86d-d23b-4da5-824e-cacea3b08ab7\") " pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" Oct 30 01:40:06.232068 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:06.216325 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cda4f86d-d23b-4da5-824e-cacea3b08ab7-istio-token\") pod \"e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg\" (UID: \"cda4f86d-d23b-4da5-824e-cacea3b08ab7\") " pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" Oct 30 01:40:06.232068 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:06.231995 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cda4f86d-d23b-4da5-824e-cacea3b08ab7-istio-podinfo\") pod \"e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg\" (UID: \"cda4f86d-d23b-4da5-824e-cacea3b08ab7\") " pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" Oct 30 01:40:06.237314 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:06.237205 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwgns\" (UniqueName: \"kubernetes.io/projected/cda4f86d-d23b-4da5-824e-cacea3b08ab7-kube-api-access-lwgns\") pod \"e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg\" (UID: \"cda4f86d-d23b-4da5-824e-cacea3b08ab7\") " pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" Oct 30 01:40:06.239069 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:06.239041 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cda4f86d-d23b-4da5-824e-cacea3b08ab7-istio-envoy\") pod \"e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg\" (UID: \"cda4f86d-d23b-4da5-824e-cacea3b08ab7\") " pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" Oct 30 01:40:06.286147 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:06.285856 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" Oct 30 01:40:06.621209 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:06.621180 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg"] Oct 30 01:40:06.626758 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:40:06.626127 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcda4f86d_d23b_4da5_824e_cacea3b08ab7.slice/crio-2044cc709ff1cba3d03d79e9d9d29445008906ee79f1398121e9941540399b19 WatchSource:0}: Error finding container 2044cc709ff1cba3d03d79e9d9d29445008906ee79f1398121e9941540399b19: Status 404 returned error can't find the container with id 2044cc709ff1cba3d03d79e9d9d29445008906ee79f1398121e9941540399b19 Oct 30 01:40:06.630367 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:06.630322 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:40:06.630486 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:06.630417 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:40:06.630486 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:06.630461 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:40:07.360667 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:07.360618 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" event={"ID":"cda4f86d-d23b-4da5-824e-cacea3b08ab7","Type":"ContainerStarted","Data":"26843365aab8f7669d8a549e01e9df0149f0c9cccb5e58dae07bc8a3061e6898"} Oct 30 01:40:07.361209 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:07.360690 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" event={"ID":"cda4f86d-d23b-4da5-824e-cacea3b08ab7","Type":"ContainerStarted","Data":"2044cc709ff1cba3d03d79e9d9d29445008906ee79f1398121e9941540399b19"} Oct 30 01:40:07.435804 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:07.435678 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" podStartSLOduration=2.435658417 podStartE2EDuration="2.435658417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:40:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:40:07.419930364 +0000 UTC m=+1330.998217700" watchObservedRunningTime="2026-10-30 01:40:07.435658417 +0000 UTC m=+1331.013945753" Oct 30 01:40:07.986476 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:07.986445 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f01cbc-5594bbb79d-bhxg9"] Oct 30 01:40:07.993923 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:07.993883 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f01cbc-5594bbb79d-bhxg9" Oct 30 01:40:08.027730 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:08.027651 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-f01cbc-serving-cert\"" Oct 30 01:40:08.043365 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:08.043328 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f01cbc-5594bbb79d-bhxg9"] Oct 30 01:40:08.172564 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:08.113720 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2416576-64de-4ee6-8082-52b1c1bb5e91-metrics-certs\") pod \"maas-api-e2e-ipp-b-f01cbc-5594bbb79d-bhxg9\" (UID: \"f2416576-64de-4ee6-8082-52b1c1bb5e91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f01cbc-5594bbb79d-bhxg9" Oct 30 01:40:08.172564 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:08.113829 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2416576-64de-4ee6-8082-52b1c1bb5e91-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f01cbc-5594bbb79d-bhxg9\" (UID: \"f2416576-64de-4ee6-8082-52b1c1bb5e91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f01cbc-5594bbb79d-bhxg9" Oct 30 01:40:08.172564 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:08.113983 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2416576-64de-4ee6-8082-52b1c1bb5e91-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f01cbc-5594bbb79d-bhxg9\" (UID: \"f2416576-64de-4ee6-8082-52b1c1bb5e91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f01cbc-5594bbb79d-bhxg9" Oct 30 01:40:08.172564 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:08.114064 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp4ls\" (UniqueName: \"kubernetes.io/projected/f2416576-64de-4ee6-8082-52b1c1bb5e91-kube-api-access-rp4ls\") pod \"maas-api-e2e-ipp-b-f01cbc-5594bbb79d-bhxg9\" (UID: \"f2416576-64de-4ee6-8082-52b1c1bb5e91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f01cbc-5594bbb79d-bhxg9" Oct 30 01:40:08.276615 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:08.228760 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp4ls\" (UniqueName: \"kubernetes.io/projected/f2416576-64de-4ee6-8082-52b1c1bb5e91-kube-api-access-rp4ls\") pod \"maas-api-e2e-ipp-b-f01cbc-5594bbb79d-bhxg9\" (UID: \"f2416576-64de-4ee6-8082-52b1c1bb5e91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f01cbc-5594bbb79d-bhxg9" Oct 30 01:40:08.276615 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:08.228875 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2416576-64de-4ee6-8082-52b1c1bb5e91-metrics-certs\") pod \"maas-api-e2e-ipp-b-f01cbc-5594bbb79d-bhxg9\" (UID: \"f2416576-64de-4ee6-8082-52b1c1bb5e91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f01cbc-5594bbb79d-bhxg9" Oct 30 01:40:08.276615 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:08.228970 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2416576-64de-4ee6-8082-52b1c1bb5e91-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f01cbc-5594bbb79d-bhxg9\" (UID: \"f2416576-64de-4ee6-8082-52b1c1bb5e91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f01cbc-5594bbb79d-bhxg9" Oct 30 01:40:08.276615 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:08.229120 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2416576-64de-4ee6-8082-52b1c1bb5e91-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f01cbc-5594bbb79d-bhxg9\" (UID: \"f2416576-64de-4ee6-8082-52b1c1bb5e91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f01cbc-5594bbb79d-bhxg9" Oct 30 01:40:08.276615 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:08.229985 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2416576-64de-4ee6-8082-52b1c1bb5e91-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f01cbc-5594bbb79d-bhxg9\" (UID: \"f2416576-64de-4ee6-8082-52b1c1bb5e91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f01cbc-5594bbb79d-bhxg9" Oct 30 01:40:08.310599 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:08.309983 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" Oct 30 01:40:08.417562 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:08.384206 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" Oct 30 01:40:08.462559 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:08.404053 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp4ls\" (UniqueName: \"kubernetes.io/projected/f2416576-64de-4ee6-8082-52b1c1bb5e91-kube-api-access-rp4ls\") pod \"maas-api-e2e-ipp-b-f01cbc-5594bbb79d-bhxg9\" (UID: \"f2416576-64de-4ee6-8082-52b1c1bb5e91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f01cbc-5594bbb79d-bhxg9" Oct 30 01:40:08.533264 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:08.493184 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" Oct 30 01:40:08.876856 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:08.876779 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2416576-64de-4ee6-8082-52b1c1bb5e91-metrics-certs\") pod \"maas-api-e2e-ipp-b-f01cbc-5594bbb79d-bhxg9\" (UID: \"f2416576-64de-4ee6-8082-52b1c1bb5e91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f01cbc-5594bbb79d-bhxg9" Oct 30 01:40:08.900115 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:08.900083 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2416576-64de-4ee6-8082-52b1c1bb5e91-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f01cbc-5594bbb79d-bhxg9\" (UID: \"f2416576-64de-4ee6-8082-52b1c1bb5e91\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f01cbc-5594bbb79d-bhxg9" Oct 30 01:40:08.903467 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:08.903439 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" Oct 30 01:40:09.111558 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:09.105779 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f01cbc-5594bbb79d-bhxg9" Oct 30 01:40:10.176789 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:10.175206 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" podUID="3b3c0e08-1371-4184-816a-d8b501526190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 30 01:40:10.176789 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:10.176482 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" podUID="ea9583aa-8259-4c9b-932e-2d6559008d86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:40:10.290314 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:10.290245 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f01cbc-5594bbb79d-bhxg9"] Oct 30 01:40:10.309561 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:40:10.309272 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2416576_64de_4ee6_8082_52b1c1bb5e91.slice/crio-2d277cf61d878b0d37c8357f93bc1194e9abf2bdbca63cb9f1028e2a6d377b2d WatchSource:0}: Error finding container 2d277cf61d878b0d37c8357f93bc1194e9abf2bdbca63cb9f1028e2a6d377b2d: Status 404 returned error can't find the container with id 2d277cf61d878b0d37c8357f93bc1194e9abf2bdbca63cb9f1028e2a6d377b2d Oct 30 01:40:10.615429 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:10.615337 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f01cbc-5594bbb79d-bhxg9" event={"ID":"f2416576-64de-4ee6-8082-52b1c1bb5e91","Type":"ContainerStarted","Data":"2d277cf61d878b0d37c8357f93bc1194e9abf2bdbca63cb9f1028e2a6d377b2d"} Oct 30 01:40:11.625082 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:11.625035 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f01cbc-5594bbb79d-bhxg9" event={"ID":"f2416576-64de-4ee6-8082-52b1c1bb5e91","Type":"ContainerStarted","Data":"44a9f54723cbb45bc882256a1ab86218a8e2220efc886d97d7039314bb001e1c"} Oct 30 01:40:11.625498 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:11.625323 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f01cbc-5594bbb79d-bhxg9" Oct 30 01:40:11.666322 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:11.666273 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f01cbc-5594bbb79d-bhxg9" podStartSLOduration=4.293476529 podStartE2EDuration="4.666260543s" totalImagesPullingTime="372.784014ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:40:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:40:11.666177502 +0000 UTC m=+1335.244464837" watchObservedRunningTime="2026-10-30 01:40:11.666260543 +0000 UTC m=+1335.244547875" Oct 30 01:40:11.812371 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:11.812333 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2ac18f-6c4f9cfb86-2gt8n"] Oct 30 01:40:11.816718 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:11.816695 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2ac18f-6c4f9cfb86-2gt8n" Oct 30 01:40:11.835704 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:11.835675 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2ac18f-6c4f9cfb86-2gt8n"] Oct 30 01:40:11.836764 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:11.836742 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2ac18f-dockercfg-gnfqr\"" Oct 30 01:40:11.837013 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:11.836991 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2ac18f\"" Oct 30 01:40:11.857266 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:11.857234 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3db79759-6455-4d30-8d81-042eaf326058-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2ac18f-6c4f9cfb86-2gt8n\" (UID: \"3db79759-6455-4d30-8d81-042eaf326058\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2ac18f-6c4f9cfb86-2gt8n" Oct 30 01:40:11.857634 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:11.857610 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g48sr\" (UniqueName: \"kubernetes.io/projected/3db79759-6455-4d30-8d81-042eaf326058-kube-api-access-g48sr\") pod \"payload-processing-e2e-ipp-a-2ac18f-6c4f9cfb86-2gt8n\" (UID: \"3db79759-6455-4d30-8d81-042eaf326058\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2ac18f-6c4f9cfb86-2gt8n" Oct 30 01:40:11.989412 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:11.989325 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g48sr\" (UniqueName: \"kubernetes.io/projected/3db79759-6455-4d30-8d81-042eaf326058-kube-api-access-g48sr\") pod \"payload-processing-e2e-ipp-a-2ac18f-6c4f9cfb86-2gt8n\" (UID: \"3db79759-6455-4d30-8d81-042eaf326058\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2ac18f-6c4f9cfb86-2gt8n" Oct 30 01:40:12.114556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:12.113826 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3db79759-6455-4d30-8d81-042eaf326058-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2ac18f-6c4f9cfb86-2gt8n\" (UID: \"3db79759-6455-4d30-8d81-042eaf326058\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2ac18f-6c4f9cfb86-2gt8n" Oct 30 01:40:12.115558 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:12.114897 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3db79759-6455-4d30-8d81-042eaf326058-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2ac18f-6c4f9cfb86-2gt8n\" (UID: \"3db79759-6455-4d30-8d81-042eaf326058\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2ac18f-6c4f9cfb86-2gt8n" Oct 30 01:40:12.164593 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:12.163042 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g48sr\" (UniqueName: \"kubernetes.io/projected/3db79759-6455-4d30-8d81-042eaf326058-kube-api-access-g48sr\") pod \"payload-processing-e2e-ipp-a-2ac18f-6c4f9cfb86-2gt8n\" (UID: \"3db79759-6455-4d30-8d81-042eaf326058\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2ac18f-6c4f9cfb86-2gt8n" Oct 30 01:40:12.430190 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:12.430156 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2ac18f-6c4f9cfb86-2gt8n" Oct 30 01:40:12.728312 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:12.728290 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2ac18f-6c4f9cfb86-2gt8n"] Oct 30 01:40:12.730524 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:40:12.730497 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3db79759_6455_4d30_8d81_042eaf326058.slice/crio-f925bff31e8aaad1a6dc5e4f82ca290790aa8f0c3ebc86efbb130a3e0220fdca WatchSource:0}: Error finding container f925bff31e8aaad1a6dc5e4f82ca290790aa8f0c3ebc86efbb130a3e0220fdca: Status 404 returned error can't find the container with id f925bff31e8aaad1a6dc5e4f82ca290790aa8f0c3ebc86efbb130a3e0220fdca Oct 30 01:40:13.638325 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:13.638282 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2ac18f-6c4f9cfb86-2gt8n" event={"ID":"3db79759-6455-4d30-8d81-042eaf326058","Type":"ContainerStarted","Data":"18743d27de0d1bfd957bc31cb1107402f124345926e8705fd8247fa9e179b99f"} Oct 30 01:40:13.638508 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:13.638349 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2ac18f-6c4f9cfb86-2gt8n" event={"ID":"3db79759-6455-4d30-8d81-042eaf326058","Type":"ContainerStarted","Data":"f925bff31e8aaad1a6dc5e4f82ca290790aa8f0c3ebc86efbb130a3e0220fdca"} Oct 30 01:40:13.638508 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:13.638383 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2ac18f-6c4f9cfb86-2gt8n" Oct 30 01:40:13.681743 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:13.681690 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-2ac18f-6c4f9cfb86-2gt8n" podStartSLOduration=2.681675981 podStartE2EDuration="2.681675981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:40:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:40:13.678678902 +0000 UTC m=+1337.256966240" watchObservedRunningTime="2026-10-30 01:40:13.681675981 +0000 UTC m=+1337.259963316" Oct 30 01:40:17.637734 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:17.637701 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f01cbc-5594bbb79d-bhxg9" Oct 30 01:40:20.693555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:20.690713 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:40:20.984596 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:20.984505 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0db534-7f9675fdc5-jr2qx"] Oct 30 01:40:21.500783 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:40:21.500739 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:40:24.648171 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:24.648141 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2ac18f-6c4f9cfb86-2gt8n" Oct 30 01:40:25.985325 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:25.985255 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db534-7f9675fdc5-jr2qx" podUID="b2f52c71-1aad-4817-b9f8-9fd8759f1961" containerName="payload-pre-processing" containerID="cri-o://177ae38f1caa01b7ccc87d3bd1f049219268dc05ba08b06f9c68059daa269c86" gracePeriod=25 Oct 30 01:40:26.266313 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:26.266288 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db534-7f9675fdc5-jr2qx" Oct 30 01:40:26.359903 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:26.359875 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2f52c71-1aad-4817-b9f8-9fd8759f1961-kube-api-access-rnc2c\" (UniqueName: \"kubernetes.io/projected/b2f52c71-1aad-4817-b9f8-9fd8759f1961-kube-api-access-rnc2c\") pod \"b2f52c71-1aad-4817-b9f8-9fd8759f1961\" (UID: \"b2f52c71-1aad-4817-b9f8-9fd8759f1961\") " Oct 30 01:40:26.360069 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:26.359909 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2f52c71-1aad-4817-b9f8-9fd8759f1961-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2f52c71-1aad-4817-b9f8-9fd8759f1961-plugins-config-volume\") pod \"b2f52c71-1aad-4817-b9f8-9fd8759f1961\" (UID: \"b2f52c71-1aad-4817-b9f8-9fd8759f1961\") " Oct 30 01:40:26.360337 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:26.360308 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2f52c71-1aad-4817-b9f8-9fd8759f1961-plugins-config-volume" pod "b2f52c71-1aad-4817-b9f8-9fd8759f1961" (UID: "b2f52c71-1aad-4817-b9f8-9fd8759f1961"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:40:26.362115 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:26.362094 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2f52c71-1aad-4817-b9f8-9fd8759f1961-kube-api-access-rnc2c" pod "b2f52c71-1aad-4817-b9f8-9fd8759f1961" (UID: "b2f52c71-1aad-4817-b9f8-9fd8759f1961"). InnerVolumeSpecName "kube-api-access-rnc2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:40:26.461521 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:26.461492 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnc2c\" (UniqueName: \"kubernetes.io/projected/b2f52c71-1aad-4817-b9f8-9fd8759f1961-kube-api-access-rnc2c\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:40:26.461521 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:26.461520 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2f52c71-1aad-4817-b9f8-9fd8759f1961-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:40:26.836874 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:26.836839 6221 generic.go:356] "Generic (PLEG): container finished" podID="b2f52c71-1aad-4817-b9f8-9fd8759f1961" containerID="177ae38f1caa01b7ccc87d3bd1f049219268dc05ba08b06f9c68059daa269c86" exitCode=0 Oct 30 01:40:26.837080 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:26.836895 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db534-7f9675fdc5-jr2qx" Oct 30 01:40:26.837080 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:26.837043 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db534-7f9675fdc5-jr2qx" event={"ID":"b2f52c71-1aad-4817-b9f8-9fd8759f1961","Type":"ContainerDied","Data":"177ae38f1caa01b7ccc87d3bd1f049219268dc05ba08b06f9c68059daa269c86"} Oct 30 01:40:26.837204 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:26.837088 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db534-7f9675fdc5-jr2qx" event={"ID":"b2f52c71-1aad-4817-b9f8-9fd8759f1961","Type":"ContainerDied","Data":"e31b95caca84e2efdf7d20a819730f7fec2a0577adc375b2eef15d1f28502602"} Oct 30 01:40:26.837204 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:26.837115 6221 scope.go:122] "RemoveContainer" containerID="177ae38f1caa01b7ccc87d3bd1f049219268dc05ba08b06f9c68059daa269c86" Oct 30 01:40:26.895554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:26.890635 6221 scope.go:122] "RemoveContainer" containerID="177ae38f1caa01b7ccc87d3bd1f049219268dc05ba08b06f9c68059daa269c86" Oct 30 01:40:26.895554 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:40:26.891078 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"177ae38f1caa01b7ccc87d3bd1f049219268dc05ba08b06f9c68059daa269c86\": container with ID starting with 177ae38f1caa01b7ccc87d3bd1f049219268dc05ba08b06f9c68059daa269c86 not found: ID does not exist" containerID="177ae38f1caa01b7ccc87d3bd1f049219268dc05ba08b06f9c68059daa269c86" Oct 30 01:40:26.895554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:26.891113 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"177ae38f1caa01b7ccc87d3bd1f049219268dc05ba08b06f9c68059daa269c86"} err="failed to get container status \"177ae38f1caa01b7ccc87d3bd1f049219268dc05ba08b06f9c68059daa269c86\": rpc error: code = NotFound desc = could not find container \"177ae38f1caa01b7ccc87d3bd1f049219268dc05ba08b06f9c68059daa269c86\": container with ID starting with 177ae38f1caa01b7ccc87d3bd1f049219268dc05ba08b06f9c68059daa269c86 not found: ID does not exist" Oct 30 01:40:26.934102 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:26.932511 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0db534-7f9675fdc5-jr2qx"] Oct 30 01:40:26.943860 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:26.943770 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0db534-7f9675fdc5-jr2qx"] Oct 30 01:40:28.565210 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:28.565165 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2f52c71-1aad-4817-b9f8-9fd8759f1961" path="/var/lib/kubelet/pods/b2f52c71-1aad-4817-b9f8-9fd8759f1961/volumes" Oct 30 01:40:39.927577 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:39.927465 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2ac18f-7bccb6bcbc-qvpb2"] Oct 30 01:40:39.928266 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:39.928247 6221 cpu_manager.go:414] "containerMap: removing container" podUID="b2f52c71-1aad-4817-b9f8-9fd8759f1961" containerName="payload-pre-processing" Oct 30 01:40:39.928324 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:39.928267 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2f52c71-1aad-4817-b9f8-9fd8759f1961" containerName="payload-pre-processing" Oct 30 01:40:39.928360 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:39.928353 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2f52c71-1aad-4817-b9f8-9fd8759f1961" containerName="payload-pre-processing" Oct 30 01:40:39.933214 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:39.933196 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2ac18f-7bccb6bcbc-qvpb2" Oct 30 01:40:39.949586 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:39.949561 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2ac18f-7bccb6bcbc-qvpb2"] Oct 30 01:40:39.960839 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:39.960811 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-2ac18f-serving-cert\"" Oct 30 01:40:40.116835 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:40.116801 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70f78ce1-27bf-4336-b601-17e028879781-metrics-certs\") pod \"maas-api-e2e-ipp-a-2ac18f-7bccb6bcbc-qvpb2\" (UID: \"70f78ce1-27bf-4336-b601-17e028879781\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2ac18f-7bccb6bcbc-qvpb2" Oct 30 01:40:40.116835 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:40.116860 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtfpx\" (UniqueName: \"kubernetes.io/projected/70f78ce1-27bf-4336-b601-17e028879781-kube-api-access-mtfpx\") pod \"maas-api-e2e-ipp-a-2ac18f-7bccb6bcbc-qvpb2\" (UID: \"70f78ce1-27bf-4336-b601-17e028879781\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2ac18f-7bccb6bcbc-qvpb2" Oct 30 01:40:40.117116 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:40.116935 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70f78ce1-27bf-4336-b601-17e028879781-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2ac18f-7bccb6bcbc-qvpb2\" (UID: \"70f78ce1-27bf-4336-b601-17e028879781\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2ac18f-7bccb6bcbc-qvpb2" Oct 30 01:40:40.117165 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:40.117133 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70f78ce1-27bf-4336-b601-17e028879781-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2ac18f-7bccb6bcbc-qvpb2\" (UID: \"70f78ce1-27bf-4336-b601-17e028879781\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2ac18f-7bccb6bcbc-qvpb2" Oct 30 01:40:40.219798 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:40.219555 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70f78ce1-27bf-4336-b601-17e028879781-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2ac18f-7bccb6bcbc-qvpb2\" (UID: \"70f78ce1-27bf-4336-b601-17e028879781\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2ac18f-7bccb6bcbc-qvpb2" Oct 30 01:40:40.219798 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:40.219742 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70f78ce1-27bf-4336-b601-17e028879781-metrics-certs\") pod \"maas-api-e2e-ipp-a-2ac18f-7bccb6bcbc-qvpb2\" (UID: \"70f78ce1-27bf-4336-b601-17e028879781\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2ac18f-7bccb6bcbc-qvpb2" Oct 30 01:40:40.220032 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:40.219813 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtfpx\" (UniqueName: \"kubernetes.io/projected/70f78ce1-27bf-4336-b601-17e028879781-kube-api-access-mtfpx\") pod \"maas-api-e2e-ipp-a-2ac18f-7bccb6bcbc-qvpb2\" (UID: \"70f78ce1-27bf-4336-b601-17e028879781\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2ac18f-7bccb6bcbc-qvpb2" Oct 30 01:40:40.220032 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:40.219911 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70f78ce1-27bf-4336-b601-17e028879781-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2ac18f-7bccb6bcbc-qvpb2\" (UID: \"70f78ce1-27bf-4336-b601-17e028879781\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2ac18f-7bccb6bcbc-qvpb2" Oct 30 01:40:40.221294 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:40.221267 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70f78ce1-27bf-4336-b601-17e028879781-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2ac18f-7bccb6bcbc-qvpb2\" (UID: \"70f78ce1-27bf-4336-b601-17e028879781\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2ac18f-7bccb6bcbc-qvpb2" Oct 30 01:40:40.240872 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:40.240839 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70f78ce1-27bf-4336-b601-17e028879781-metrics-certs\") pod \"maas-api-e2e-ipp-a-2ac18f-7bccb6bcbc-qvpb2\" (UID: \"70f78ce1-27bf-4336-b601-17e028879781\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2ac18f-7bccb6bcbc-qvpb2" Oct 30 01:40:40.247629 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:40.245076 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70f78ce1-27bf-4336-b601-17e028879781-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2ac18f-7bccb6bcbc-qvpb2\" (UID: \"70f78ce1-27bf-4336-b601-17e028879781\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2ac18f-7bccb6bcbc-qvpb2" Oct 30 01:40:40.297566 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:40.295221 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtfpx\" (UniqueName: \"kubernetes.io/projected/70f78ce1-27bf-4336-b601-17e028879781-kube-api-access-mtfpx\") pod \"maas-api-e2e-ipp-a-2ac18f-7bccb6bcbc-qvpb2\" (UID: \"70f78ce1-27bf-4336-b601-17e028879781\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2ac18f-7bccb6bcbc-qvpb2" Oct 30 01:40:40.546610 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:40.546579 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2ac18f-7bccb6bcbc-qvpb2" Oct 30 01:40:40.871747 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:40.871412 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2ac18f-7bccb6bcbc-qvpb2"] Oct 30 01:40:40.875368 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:40:40.875328 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70f78ce1_27bf_4336_b601_17e028879781.slice/crio-383e4c94f00acd66fd172b68bf341f53297f3df0ecc9c099a2ee28288126805a WatchSource:0}: Error finding container 383e4c94f00acd66fd172b68bf341f53297f3df0ecc9c099a2ee28288126805a: Status 404 returned error can't find the container with id 383e4c94f00acd66fd172b68bf341f53297f3df0ecc9c099a2ee28288126805a Oct 30 01:40:40.957346 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:40.957303 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2ac18f-7bccb6bcbc-qvpb2" event={"ID":"70f78ce1-27bf-4336-b601-17e028879781","Type":"ContainerStarted","Data":"383e4c94f00acd66fd172b68bf341f53297f3df0ecc9c099a2ee28288126805a"} Oct 30 01:40:41.965626 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:41.965589 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2ac18f-7bccb6bcbc-qvpb2" event={"ID":"70f78ce1-27bf-4336-b601-17e028879781","Type":"ContainerStarted","Data":"1992f6d5b264d191c0a4d6e9301195829422805c2e1a06bbbe3e7f88fd7d3206"} Oct 30 01:40:41.966028 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:41.965732 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2ac18f-7bccb6bcbc-qvpb2" Oct 30 01:40:42.010838 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:42.010788 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2ac18f-7bccb6bcbc-qvpb2" podStartSLOduration=2.482406702 podStartE2EDuration="3.010770805s" totalImagesPullingTime="528.364103ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:40:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:40:42.008524573 +0000 UTC m=+1365.586811908" watchObservedRunningTime="2026-10-30 01:40:42.010770805 +0000 UTC m=+1365.589058141" Oct 30 01:40:42.453877 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:42.453845 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f01cbc-76b448f6-jkd4f"] Oct 30 01:40:42.489059 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:42.489026 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f01cbc-664df47bc-pgtv2"] Oct 30 01:40:44.924969 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:44.924944 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-wp6mf_fc46a03a-6581-4d32-8ed7-a4d5c53625b4/payload-processing/0.log" Oct 30 01:40:45.137825 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:45.137798 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-wp6mf_fc46a03a-6581-4d32-8ed7-a4d5c53625b4/payload-processing/0.log" Oct 30 01:40:47.454524 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:47.454483 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f01cbc-76b448f6-jkd4f" podUID="d0a4deee-d48d-4533-a043-34d7ca72705f" containerName="payload-processing" containerID="cri-o://3c6eef4a00d9ddcd64f8953e5f2fedc6b41420759864504e8389b3ea13496c61" gracePeriod=25 Oct 30 01:40:47.490203 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:47.490146 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f01cbc-664df47bc-pgtv2" podUID="c7c2b8d5-5ea1-4ddd-8ad9-fd98031a9f8c" containerName="payload-pre-processing" containerID="cri-o://78b6e9e4d99b38c3a133faf49dcc271e79d362385be46a2d5521e2036cf448fd" gracePeriod=25 Oct 30 01:40:47.776758 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:47.776731 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f01cbc-76b448f6-jkd4f" Oct 30 01:40:47.817048 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:47.817021 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f01cbc-664df47bc-pgtv2" Oct 30 01:40:47.890555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:47.890470 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0a4deee-d48d-4533-a043-34d7ca72705f-kube-api-access-n9msd\" (UniqueName: \"kubernetes.io/projected/d0a4deee-d48d-4533-a043-34d7ca72705f-kube-api-access-n9msd\") pod \"d0a4deee-d48d-4533-a043-34d7ca72705f\" (UID: \"d0a4deee-d48d-4533-a043-34d7ca72705f\") " Oct 30 01:40:47.890743 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:47.890591 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7c2b8d5-5ea1-4ddd-8ad9-fd98031a9f8c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7c2b8d5-5ea1-4ddd-8ad9-fd98031a9f8c-plugins-config-volume\") pod \"c7c2b8d5-5ea1-4ddd-8ad9-fd98031a9f8c\" (UID: \"c7c2b8d5-5ea1-4ddd-8ad9-fd98031a9f8c\") " Oct 30 01:40:47.890743 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:47.890617 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7c2b8d5-5ea1-4ddd-8ad9-fd98031a9f8c-kube-api-access-hcpb4\" (UniqueName: \"kubernetes.io/projected/c7c2b8d5-5ea1-4ddd-8ad9-fd98031a9f8c-kube-api-access-hcpb4\") pod \"c7c2b8d5-5ea1-4ddd-8ad9-fd98031a9f8c\" (UID: \"c7c2b8d5-5ea1-4ddd-8ad9-fd98031a9f8c\") " Oct 30 01:40:47.890743 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:47.890683 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0a4deee-d48d-4533-a043-34d7ca72705f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0a4deee-d48d-4533-a043-34d7ca72705f-plugins-config-volume\") pod \"d0a4deee-d48d-4533-a043-34d7ca72705f\" (UID: \"d0a4deee-d48d-4533-a043-34d7ca72705f\") " Oct 30 01:40:47.911561 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:47.891786 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0a4deee-d48d-4533-a043-34d7ca72705f-plugins-config-volume" pod "d0a4deee-d48d-4533-a043-34d7ca72705f" (UID: "d0a4deee-d48d-4533-a043-34d7ca72705f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:40:47.954980 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:47.954667 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7c2b8d5-5ea1-4ddd-8ad9-fd98031a9f8c-plugins-config-volume" pod "c7c2b8d5-5ea1-4ddd-8ad9-fd98031a9f8c" (UID: "c7c2b8d5-5ea1-4ddd-8ad9-fd98031a9f8c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:40:47.961771 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:47.961672 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7c2b8d5-5ea1-4ddd-8ad9-fd98031a9f8c-kube-api-access-hcpb4" pod "c7c2b8d5-5ea1-4ddd-8ad9-fd98031a9f8c" (UID: "c7c2b8d5-5ea1-4ddd-8ad9-fd98031a9f8c"). InnerVolumeSpecName "kube-api-access-hcpb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:40:47.996802 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:47.993246 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0a4deee-d48d-4533-a043-34d7ca72705f-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:40:47.996802 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:47.993283 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7c2b8d5-5ea1-4ddd-8ad9-fd98031a9f8c-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:40:47.996802 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:47.993293 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcpb4\" (UniqueName: \"kubernetes.io/projected/c7c2b8d5-5ea1-4ddd-8ad9-fd98031a9f8c-kube-api-access-hcpb4\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:40:47.997196 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:47.997146 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0a4deee-d48d-4533-a043-34d7ca72705f-kube-api-access-n9msd" pod "d0a4deee-d48d-4533-a043-34d7ca72705f" (UID: "d0a4deee-d48d-4533-a043-34d7ca72705f"). InnerVolumeSpecName "kube-api-access-n9msd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:40:48.026439 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:48.026410 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2ac18f-7bccb6bcbc-qvpb2" Oct 30 01:40:48.028407 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:48.028378 6221 generic.go:356] "Generic (PLEG): container finished" podID="c7c2b8d5-5ea1-4ddd-8ad9-fd98031a9f8c" containerID="78b6e9e4d99b38c3a133faf49dcc271e79d362385be46a2d5521e2036cf448fd" exitCode=0 Oct 30 01:40:48.028589 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:48.028569 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f01cbc-664df47bc-pgtv2" event={"ID":"c7c2b8d5-5ea1-4ddd-8ad9-fd98031a9f8c","Type":"ContainerDied","Data":"78b6e9e4d99b38c3a133faf49dcc271e79d362385be46a2d5521e2036cf448fd"} Oct 30 01:40:48.028801 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:48.028618 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f01cbc-664df47bc-pgtv2" event={"ID":"c7c2b8d5-5ea1-4ddd-8ad9-fd98031a9f8c","Type":"ContainerDied","Data":"dfd68e29a358ee0093be9e6b04d02857db235a91cffccf085523bfe32bd9352a"} Oct 30 01:40:48.028801 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:48.028641 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f01cbc-664df47bc-pgtv2" Oct 30 01:40:48.028801 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:48.028648 6221 scope.go:122] "RemoveContainer" containerID="78b6e9e4d99b38c3a133faf49dcc271e79d362385be46a2d5521e2036cf448fd" Oct 30 01:40:48.032069 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:48.032042 6221 generic.go:356] "Generic (PLEG): container finished" podID="d0a4deee-d48d-4533-a043-34d7ca72705f" containerID="3c6eef4a00d9ddcd64f8953e5f2fedc6b41420759864504e8389b3ea13496c61" exitCode=0 Oct 30 01:40:48.032165 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:48.032111 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f01cbc-76b448f6-jkd4f" event={"ID":"d0a4deee-d48d-4533-a043-34d7ca72705f","Type":"ContainerDied","Data":"3c6eef4a00d9ddcd64f8953e5f2fedc6b41420759864504e8389b3ea13496c61"} Oct 30 01:40:48.032165 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:48.032154 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f01cbc-76b448f6-jkd4f" event={"ID":"d0a4deee-d48d-4533-a043-34d7ca72705f","Type":"ContainerDied","Data":"e77e53bf19378cd463d140fe50081ab9a45f2ef401211d10507dfcc3bc653d29"} Oct 30 01:40:48.032279 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:48.032239 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f01cbc-76b448f6-jkd4f" Oct 30 01:40:48.044281 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:48.044262 6221 scope.go:122] "RemoveContainer" containerID="78b6e9e4d99b38c3a133faf49dcc271e79d362385be46a2d5521e2036cf448fd" Oct 30 01:40:48.044599 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:40:48.044567 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78b6e9e4d99b38c3a133faf49dcc271e79d362385be46a2d5521e2036cf448fd\": container with ID starting with 78b6e9e4d99b38c3a133faf49dcc271e79d362385be46a2d5521e2036cf448fd not found: ID does not exist" containerID="78b6e9e4d99b38c3a133faf49dcc271e79d362385be46a2d5521e2036cf448fd" Oct 30 01:40:48.044693 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:48.044602 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78b6e9e4d99b38c3a133faf49dcc271e79d362385be46a2d5521e2036cf448fd"} err="failed to get container status \"78b6e9e4d99b38c3a133faf49dcc271e79d362385be46a2d5521e2036cf448fd\": rpc error: code = NotFound desc = could not find container \"78b6e9e4d99b38c3a133faf49dcc271e79d362385be46a2d5521e2036cf448fd\": container with ID starting with 78b6e9e4d99b38c3a133faf49dcc271e79d362385be46a2d5521e2036cf448fd not found: ID does not exist" Oct 30 01:40:48.044693 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:48.044620 6221 scope.go:122] "RemoveContainer" containerID="3c6eef4a00d9ddcd64f8953e5f2fedc6b41420759864504e8389b3ea13496c61" Oct 30 01:40:48.053521 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:48.053501 6221 scope.go:122] "RemoveContainer" containerID="3c6eef4a00d9ddcd64f8953e5f2fedc6b41420759864504e8389b3ea13496c61" Oct 30 01:40:48.053827 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:40:48.053809 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c6eef4a00d9ddcd64f8953e5f2fedc6b41420759864504e8389b3ea13496c61\": container with ID starting with 3c6eef4a00d9ddcd64f8953e5f2fedc6b41420759864504e8389b3ea13496c61 not found: ID does not exist" containerID="3c6eef4a00d9ddcd64f8953e5f2fedc6b41420759864504e8389b3ea13496c61" Oct 30 01:40:48.053873 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:48.053833 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c6eef4a00d9ddcd64f8953e5f2fedc6b41420759864504e8389b3ea13496c61"} err="failed to get container status \"3c6eef4a00d9ddcd64f8953e5f2fedc6b41420759864504e8389b3ea13496c61\": rpc error: code = NotFound desc = could not find container \"3c6eef4a00d9ddcd64f8953e5f2fedc6b41420759864504e8389b3ea13496c61\": container with ID starting with 3c6eef4a00d9ddcd64f8953e5f2fedc6b41420759864504e8389b3ea13496c61 not found: ID does not exist" Oct 30 01:40:48.095495 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:48.095457 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9msd\" (UniqueName: \"kubernetes.io/projected/d0a4deee-d48d-4533-a043-34d7ca72705f-kube-api-access-n9msd\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:40:48.108121 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:48.108086 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f01cbc-664df47bc-pgtv2"] Oct 30 01:40:48.111835 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:48.111117 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f01cbc-664df47bc-pgtv2"] Oct 30 01:40:48.142992 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:48.142963 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0db534-77cccfc547-jh5tl"] Oct 30 01:40:48.143707 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:48.143685 6221 cpu_manager.go:414] "containerMap: removing container" podUID="d0a4deee-d48d-4533-a043-34d7ca72705f" containerName="payload-processing" Oct 30 01:40:48.143707 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:48.143707 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0a4deee-d48d-4533-a043-34d7ca72705f" containerName="payload-processing" Oct 30 01:40:48.143902 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:48.143730 6221 cpu_manager.go:414] "containerMap: removing container" podUID="c7c2b8d5-5ea1-4ddd-8ad9-fd98031a9f8c" containerName="payload-pre-processing" Oct 30 01:40:48.143902 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:48.143738 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7c2b8d5-5ea1-4ddd-8ad9-fd98031a9f8c" containerName="payload-pre-processing" Oct 30 01:40:48.144004 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:48.143912 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0a4deee-d48d-4533-a043-34d7ca72705f" containerName="payload-processing" Oct 30 01:40:48.144004 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:48.143931 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7c2b8d5-5ea1-4ddd-8ad9-fd98031a9f8c" containerName="payload-pre-processing" Oct 30 01:40:48.148507 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:48.148487 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f01cbc-76b448f6-jkd4f"] Oct 30 01:40:48.148621 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:48.148601 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db534-77cccfc547-jh5tl" Oct 30 01:40:48.154110 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:48.154090 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f01cbc-76b448f6-jkd4f"] Oct 30 01:40:48.164087 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:48.164067 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0db534-77cccfc547-jh5tl"] Oct 30 01:40:48.165587 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:48.165558 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-0db534\"" Oct 30 01:40:48.165669 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:48.165652 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-0db534-dockercfg-h29p2\"" Oct 30 01:40:48.197046 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:48.197013 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5271795-2ae6-4a9e-a49c-dc6f30cbf180-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0db534-77cccfc547-jh5tl\" (UID: \"c5271795-2ae6-4a9e-a49c-dc6f30cbf180\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db534-77cccfc547-jh5tl" Oct 30 01:40:48.197225 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:48.197059 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67jk8\" (UniqueName: \"kubernetes.io/projected/c5271795-2ae6-4a9e-a49c-dc6f30cbf180-kube-api-access-67jk8\") pod \"payload-pre-processing-e2e-ipp-b-0db534-77cccfc547-jh5tl\" (UID: \"c5271795-2ae6-4a9e-a49c-dc6f30cbf180\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db534-77cccfc547-jh5tl" Oct 30 01:40:48.318558 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:48.299421 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67jk8\" (UniqueName: \"kubernetes.io/projected/c5271795-2ae6-4a9e-a49c-dc6f30cbf180-kube-api-access-67jk8\") pod \"payload-pre-processing-e2e-ipp-b-0db534-77cccfc547-jh5tl\" (UID: \"c5271795-2ae6-4a9e-a49c-dc6f30cbf180\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db534-77cccfc547-jh5tl" Oct 30 01:40:48.318558 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:48.299806 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5271795-2ae6-4a9e-a49c-dc6f30cbf180-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0db534-77cccfc547-jh5tl\" (UID: \"c5271795-2ae6-4a9e-a49c-dc6f30cbf180\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db534-77cccfc547-jh5tl" Oct 30 01:40:48.318558 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:48.300899 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5271795-2ae6-4a9e-a49c-dc6f30cbf180-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-0db534-77cccfc547-jh5tl\" (UID: \"c5271795-2ae6-4a9e-a49c-dc6f30cbf180\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db534-77cccfc547-jh5tl" Oct 30 01:40:48.334582 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:48.333652 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67jk8\" (UniqueName: \"kubernetes.io/projected/c5271795-2ae6-4a9e-a49c-dc6f30cbf180-kube-api-access-67jk8\") pod \"payload-pre-processing-e2e-ipp-b-0db534-77cccfc547-jh5tl\" (UID: \"c5271795-2ae6-4a9e-a49c-dc6f30cbf180\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db534-77cccfc547-jh5tl" Oct 30 01:40:48.466588 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:48.462675 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db534-77cccfc547-jh5tl" Oct 30 01:40:48.553047 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:48.551939 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7c2b8d5-5ea1-4ddd-8ad9-fd98031a9f8c" path="/var/lib/kubelet/pods/c7c2b8d5-5ea1-4ddd-8ad9-fd98031a9f8c/volumes" Oct 30 01:40:48.553047 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:48.552343 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0a4deee-d48d-4533-a043-34d7ca72705f" path="/var/lib/kubelet/pods/d0a4deee-d48d-4533-a043-34d7ca72705f/volumes" Oct 30 01:40:48.768366 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:48.768338 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0db534-77cccfc547-jh5tl"] Oct 30 01:40:48.770953 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:40:48.770925 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5271795_2ae6_4a9e_a49c_dc6f30cbf180.slice/crio-f5437f18b0cd68a19c22eeb55f89eb519b0fac0dfa24fe085f98c616853b48d0 WatchSource:0}: Error finding container f5437f18b0cd68a19c22eeb55f89eb519b0fac0dfa24fe085f98c616853b48d0: Status 404 returned error can't find the container with id f5437f18b0cd68a19c22eeb55f89eb519b0fac0dfa24fe085f98c616853b48d0 Oct 30 01:40:49.052086 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:49.052054 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db534-77cccfc547-jh5tl" event={"ID":"c5271795-2ae6-4a9e-a49c-dc6f30cbf180","Type":"ContainerStarted","Data":"f5437f18b0cd68a19c22eeb55f89eb519b0fac0dfa24fe085f98c616853b48d0"} Oct 30 01:40:50.653803 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:50.653768 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-6b3dbd/ipp-route-6b3dbd-kserve-894c568b9-d6fnn"] Oct 30 01:40:50.657986 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:50.657962 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-6b3dbd/ipp-route-6b3dbd-kserve-894c568b9-d6fnn" Oct 30 01:40:50.693584 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:50.689221 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-6b3dbd\"/\"default-dockercfg-kfrxj\"" Oct 30 01:40:50.693584 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:50.689354 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-6b3dbd\"/\"ipp-route-6b3dbd-kserve-self-signed-certs\"" Oct 30 01:40:50.693584 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:50.689424 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-6b3dbd\"/\"kube-root-ca.crt\"" Oct 30 01:40:50.693584 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:50.689496 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-6b3dbd\"/\"openshift-service-ca.crt\"" Oct 30 01:40:50.693584 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:50.691716 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-6b3dbd/ipp-route-6b3dbd-kserve-894c568b9-d6fnn"] Oct 30 01:40:50.739682 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:50.739601 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa2ee40b-7d02-4d12-9894-7d4f51053320-home\") pod \"ipp-route-6b3dbd-kserve-894c568b9-d6fnn\" (UID: \"aa2ee40b-7d02-4d12-9894-7d4f51053320\") " pod="ai-tenant-e2e-ipp-a-6b3dbd/ipp-route-6b3dbd-kserve-894c568b9-d6fnn" Oct 30 01:40:50.739839 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:50.739715 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa2ee40b-7d02-4d12-9894-7d4f51053320-tls-certs\") pod \"ipp-route-6b3dbd-kserve-894c568b9-d6fnn\" (UID: \"aa2ee40b-7d02-4d12-9894-7d4f51053320\") " pod="ai-tenant-e2e-ipp-a-6b3dbd/ipp-route-6b3dbd-kserve-894c568b9-d6fnn" Oct 30 01:40:50.739839 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:50.739765 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa2ee40b-7d02-4d12-9894-7d4f51053320-model-cache\") pod \"ipp-route-6b3dbd-kserve-894c568b9-d6fnn\" (UID: \"aa2ee40b-7d02-4d12-9894-7d4f51053320\") " pod="ai-tenant-e2e-ipp-a-6b3dbd/ipp-route-6b3dbd-kserve-894c568b9-d6fnn" Oct 30 01:40:50.740033 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:50.740012 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa2ee40b-7d02-4d12-9894-7d4f51053320-dshm\") pod \"ipp-route-6b3dbd-kserve-894c568b9-d6fnn\" (UID: \"aa2ee40b-7d02-4d12-9894-7d4f51053320\") " pod="ai-tenant-e2e-ipp-a-6b3dbd/ipp-route-6b3dbd-kserve-894c568b9-d6fnn" Oct 30 01:40:50.740091 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:50.740049 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa2ee40b-7d02-4d12-9894-7d4f51053320-tmp-dir\") pod \"ipp-route-6b3dbd-kserve-894c568b9-d6fnn\" (UID: \"aa2ee40b-7d02-4d12-9894-7d4f51053320\") " pod="ai-tenant-e2e-ipp-a-6b3dbd/ipp-route-6b3dbd-kserve-894c568b9-d6fnn" Oct 30 01:40:50.740145 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:50.740110 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr7nt\" (UniqueName: \"kubernetes.io/projected/aa2ee40b-7d02-4d12-9894-7d4f51053320-kube-api-access-nr7nt\") pod \"ipp-route-6b3dbd-kserve-894c568b9-d6fnn\" (UID: \"aa2ee40b-7d02-4d12-9894-7d4f51053320\") " pod="ai-tenant-e2e-ipp-a-6b3dbd/ipp-route-6b3dbd-kserve-894c568b9-d6fnn" Oct 30 01:40:50.841840 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:50.841803 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa2ee40b-7d02-4d12-9894-7d4f51053320-tls-certs\") pod \"ipp-route-6b3dbd-kserve-894c568b9-d6fnn\" (UID: \"aa2ee40b-7d02-4d12-9894-7d4f51053320\") " pod="ai-tenant-e2e-ipp-a-6b3dbd/ipp-route-6b3dbd-kserve-894c568b9-d6fnn" Oct 30 01:40:50.842044 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:50.841854 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa2ee40b-7d02-4d12-9894-7d4f51053320-model-cache\") pod \"ipp-route-6b3dbd-kserve-894c568b9-d6fnn\" (UID: \"aa2ee40b-7d02-4d12-9894-7d4f51053320\") " pod="ai-tenant-e2e-ipp-a-6b3dbd/ipp-route-6b3dbd-kserve-894c568b9-d6fnn" Oct 30 01:40:50.842044 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:50.842015 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa2ee40b-7d02-4d12-9894-7d4f51053320-dshm\") pod \"ipp-route-6b3dbd-kserve-894c568b9-d6fnn\" (UID: \"aa2ee40b-7d02-4d12-9894-7d4f51053320\") " pod="ai-tenant-e2e-ipp-a-6b3dbd/ipp-route-6b3dbd-kserve-894c568b9-d6fnn" Oct 30 01:40:50.842151 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:50.842042 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa2ee40b-7d02-4d12-9894-7d4f51053320-tmp-dir\") pod \"ipp-route-6b3dbd-kserve-894c568b9-d6fnn\" (UID: \"aa2ee40b-7d02-4d12-9894-7d4f51053320\") " pod="ai-tenant-e2e-ipp-a-6b3dbd/ipp-route-6b3dbd-kserve-894c568b9-d6fnn" Oct 30 01:40:50.842151 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:50.842064 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr7nt\" (UniqueName: \"kubernetes.io/projected/aa2ee40b-7d02-4d12-9894-7d4f51053320-kube-api-access-nr7nt\") pod \"ipp-route-6b3dbd-kserve-894c568b9-d6fnn\" (UID: \"aa2ee40b-7d02-4d12-9894-7d4f51053320\") " pod="ai-tenant-e2e-ipp-a-6b3dbd/ipp-route-6b3dbd-kserve-894c568b9-d6fnn" Oct 30 01:40:50.842247 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:50.842155 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa2ee40b-7d02-4d12-9894-7d4f51053320-home\") pod \"ipp-route-6b3dbd-kserve-894c568b9-d6fnn\" (UID: \"aa2ee40b-7d02-4d12-9894-7d4f51053320\") " pod="ai-tenant-e2e-ipp-a-6b3dbd/ipp-route-6b3dbd-kserve-894c568b9-d6fnn" Oct 30 01:40:50.842387 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:50.842361 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa2ee40b-7d02-4d12-9894-7d4f51053320-model-cache\") pod \"ipp-route-6b3dbd-kserve-894c568b9-d6fnn\" (UID: \"aa2ee40b-7d02-4d12-9894-7d4f51053320\") " pod="ai-tenant-e2e-ipp-a-6b3dbd/ipp-route-6b3dbd-kserve-894c568b9-d6fnn" Oct 30 01:40:50.842524 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:50.842426 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa2ee40b-7d02-4d12-9894-7d4f51053320-tmp-dir\") pod \"ipp-route-6b3dbd-kserve-894c568b9-d6fnn\" (UID: \"aa2ee40b-7d02-4d12-9894-7d4f51053320\") " pod="ai-tenant-e2e-ipp-a-6b3dbd/ipp-route-6b3dbd-kserve-894c568b9-d6fnn" Oct 30 01:40:50.842626 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:50.842453 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa2ee40b-7d02-4d12-9894-7d4f51053320-home\") pod \"ipp-route-6b3dbd-kserve-894c568b9-d6fnn\" (UID: \"aa2ee40b-7d02-4d12-9894-7d4f51053320\") " pod="ai-tenant-e2e-ipp-a-6b3dbd/ipp-route-6b3dbd-kserve-894c568b9-d6fnn" Oct 30 01:40:50.844725 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:50.844705 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa2ee40b-7d02-4d12-9894-7d4f51053320-dshm\") pod \"ipp-route-6b3dbd-kserve-894c568b9-d6fnn\" (UID: \"aa2ee40b-7d02-4d12-9894-7d4f51053320\") " pod="ai-tenant-e2e-ipp-a-6b3dbd/ipp-route-6b3dbd-kserve-894c568b9-d6fnn" Oct 30 01:40:50.844985 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:50.844966 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa2ee40b-7d02-4d12-9894-7d4f51053320-tls-certs\") pod \"ipp-route-6b3dbd-kserve-894c568b9-d6fnn\" (UID: \"aa2ee40b-7d02-4d12-9894-7d4f51053320\") " pod="ai-tenant-e2e-ipp-a-6b3dbd/ipp-route-6b3dbd-kserve-894c568b9-d6fnn" Oct 30 01:40:50.861883 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:50.861856 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr7nt\" (UniqueName: \"kubernetes.io/projected/aa2ee40b-7d02-4d12-9894-7d4f51053320-kube-api-access-nr7nt\") pod \"ipp-route-6b3dbd-kserve-894c568b9-d6fnn\" (UID: \"aa2ee40b-7d02-4d12-9894-7d4f51053320\") " pod="ai-tenant-e2e-ipp-a-6b3dbd/ipp-route-6b3dbd-kserve-894c568b9-d6fnn" Oct 30 01:40:50.987557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:50.984970 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-6b3dbd/ipp-route-6b3dbd-kserve-894c568b9-d6fnn" Oct 30 01:40:51.069921 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:51.069805 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db534-77cccfc547-jh5tl" event={"ID":"c5271795-2ae6-4a9e-a49c-dc6f30cbf180","Type":"ContainerStarted","Data":"19a4848c6d399476f6d553db5f523a7671c7bdce20db58053e8b5075f9160f42"} Oct 30 01:40:51.070079 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:51.070063 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db534-77cccfc547-jh5tl" Oct 30 01:40:51.390411 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:51.390357 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db534-77cccfc547-jh5tl" podStartSLOduration=1.677412305 podStartE2EDuration="3.390340661s" totalImagesPullingTime="1.712928356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:40:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:40:51.11083058 +0000 UTC m=+1374.689117929" watchObservedRunningTime="2026-10-30 01:40:51.390340661 +0000 UTC m=+1374.968627997" Oct 30 01:40:51.390853 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:51.390828 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-6b3dbd/ipp-route-6b3dbd-kserve-894c568b9-d6fnn"] Oct 30 01:40:51.393896 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:40:51.393872 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa2ee40b_7d02_4d12_9894_7d4f51053320.slice/crio-ca293ce1c0eec6d8b28e93f45223d0d2511bff190ccb507bfd566fae2cbd2869 WatchSource:0}: Error finding container ca293ce1c0eec6d8b28e93f45223d0d2511bff190ccb507bfd566fae2cbd2869: Status 404 returned error can't find the container with id ca293ce1c0eec6d8b28e93f45223d0d2511bff190ccb507bfd566fae2cbd2869 Oct 30 01:40:51.499499 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:40:51.499454 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:40:52.077751 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:52.077709 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-6b3dbd/ipp-route-6b3dbd-kserve-894c568b9-d6fnn" event={"ID":"aa2ee40b-7d02-4d12-9894-7d4f51053320","Type":"ContainerStarted","Data":"62e967e4891328ea6d73d068750ac60c84669322ee17d8bb832ad3d6b678486f"} Oct 30 01:40:52.077751 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:52.077754 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-6b3dbd/ipp-route-6b3dbd-kserve-894c568b9-d6fnn" event={"ID":"aa2ee40b-7d02-4d12-9894-7d4f51053320","Type":"ContainerStarted","Data":"ca293ce1c0eec6d8b28e93f45223d0d2511bff190ccb507bfd566fae2cbd2869"} Oct 30 01:40:52.119944 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:52.119883 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-6b3dbd/ipp-route-6b3dbd-kserve-894c568b9-d6fnn" podStartSLOduration=2.119866138 podStartE2EDuration="2.119866138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:40:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:40:52.118848313 +0000 UTC m=+1375.697135663" watchObservedRunningTime="2026-10-30 01:40:52.119866138 +0000 UTC m=+1375.698153476" Oct 30 01:40:52.985775 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:52.985747 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-6b3dbd/ipp-route-6b3dbd-kserve-894c568b9-d6fnn" Oct 30 01:40:55.481541 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:55.481497 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:40:59.297408 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:59.297372 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-19a5f1-56c7d78bc7-4bp7l"] Oct 30 01:40:59.297987 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:59.297647 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-19a5f1-56c7d78bc7-4bp7l" podUID="a2eebf2d-067e-4524-a8d2-eb97208c89e4" containerName="maas-api" containerID="cri-o://0d8a364542cd5799bc8ae15a8ef9827aad96c68962f0aacfd63ee7e71b641f76" gracePeriod=30 Oct 30 01:40:59.681095 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:59.681072 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-19a5f1-56c7d78bc7-4bp7l" Oct 30 01:40:59.857936 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:59.857881 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2eebf2d-067e-4524-a8d2-eb97208c89e4-kube-api-access-s525f\" (UniqueName: \"kubernetes.io/projected/a2eebf2d-067e-4524-a8d2-eb97208c89e4-kube-api-access-s525f\") pod \"a2eebf2d-067e-4524-a8d2-eb97208c89e4\" (UID: \"a2eebf2d-067e-4524-a8d2-eb97208c89e4\") " Oct 30 01:40:59.859190 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:59.859140 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2eebf2d-067e-4524-a8d2-eb97208c89e4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2eebf2d-067e-4524-a8d2-eb97208c89e4-openshift-service-ca\") pod \"a2eebf2d-067e-4524-a8d2-eb97208c89e4\" (UID: \"a2eebf2d-067e-4524-a8d2-eb97208c89e4\") " Oct 30 01:40:59.860111 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:59.859968 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2eebf2d-067e-4524-a8d2-eb97208c89e4-openshift-service-ca" pod "a2eebf2d-067e-4524-a8d2-eb97208c89e4" (UID: "a2eebf2d-067e-4524-a8d2-eb97208c89e4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:40:59.860321 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:59.860286 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2eebf2d-067e-4524-a8d2-eb97208c89e4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2eebf2d-067e-4524-a8d2-eb97208c89e4-maas-api-tls\") pod \"a2eebf2d-067e-4524-a8d2-eb97208c89e4\" (UID: \"a2eebf2d-067e-4524-a8d2-eb97208c89e4\") " Oct 30 01:40:59.893155 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:59.860840 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2eebf2d-067e-4524-a8d2-eb97208c89e4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2eebf2d-067e-4524-a8d2-eb97208c89e4-metrics-certs\") pod \"a2eebf2d-067e-4524-a8d2-eb97208c89e4\" (UID: \"a2eebf2d-067e-4524-a8d2-eb97208c89e4\") " Oct 30 01:40:59.893512 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:59.882126 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2eebf2d-067e-4524-a8d2-eb97208c89e4-metrics-certs" pod "a2eebf2d-067e-4524-a8d2-eb97208c89e4" (UID: "a2eebf2d-067e-4524-a8d2-eb97208c89e4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:40:59.922758 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:59.922726 6221 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2eebf2d-067e-4524-a8d2-eb97208c89e4-metrics-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:40:59.930695 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:40:59.930664 6221 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2eebf2d-067e-4524-a8d2-eb97208c89e4-openshift-service-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:41:00.045026 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:00.044985 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2eebf2d-067e-4524-a8d2-eb97208c89e4-kube-api-access-s525f" pod "a2eebf2d-067e-4524-a8d2-eb97208c89e4" (UID: "a2eebf2d-067e-4524-a8d2-eb97208c89e4"). InnerVolumeSpecName "kube-api-access-s525f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:41:00.056587 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:00.045710 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s525f\" (UniqueName: \"kubernetes.io/projected/a2eebf2d-067e-4524-a8d2-eb97208c89e4-kube-api-access-s525f\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:41:00.079313 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:00.079223 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2eebf2d-067e-4524-a8d2-eb97208c89e4-maas-api-tls" pod "a2eebf2d-067e-4524-a8d2-eb97208c89e4" (UID: "a2eebf2d-067e-4524-a8d2-eb97208c89e4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:41:00.152951 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:00.152916 6221 generic.go:356] "Generic (PLEG): container finished" podID="a2eebf2d-067e-4524-a8d2-eb97208c89e4" containerID="0d8a364542cd5799bc8ae15a8ef9827aad96c68962f0aacfd63ee7e71b641f76" exitCode=0 Oct 30 01:41:00.153141 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:00.152971 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-19a5f1-56c7d78bc7-4bp7l" event={"ID":"a2eebf2d-067e-4524-a8d2-eb97208c89e4","Type":"ContainerDied","Data":"0d8a364542cd5799bc8ae15a8ef9827aad96c68962f0aacfd63ee7e71b641f76"} Oct 30 01:41:00.153141 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:00.153048 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-19a5f1-56c7d78bc7-4bp7l" event={"ID":"a2eebf2d-067e-4524-a8d2-eb97208c89e4","Type":"ContainerDied","Data":"92bb68460188fe54781c1d44bd37ba5e8deb73e03e60e6c820194c2afa1ceaa1"} Oct 30 01:41:00.153141 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:00.153079 6221 scope.go:122] "RemoveContainer" containerID="0d8a364542cd5799bc8ae15a8ef9827aad96c68962f0aacfd63ee7e71b641f76" Oct 30 01:41:00.153321 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:00.153240 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-19a5f1-56c7d78bc7-4bp7l" Oct 30 01:41:00.157842 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:00.157809 6221 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2eebf2d-067e-4524-a8d2-eb97208c89e4-maas-api-tls\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:41:00.165306 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:00.165290 6221 scope.go:122] "RemoveContainer" containerID="0d8a364542cd5799bc8ae15a8ef9827aad96c68962f0aacfd63ee7e71b641f76" Oct 30 01:41:00.165643 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:41:00.165621 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d8a364542cd5799bc8ae15a8ef9827aad96c68962f0aacfd63ee7e71b641f76\": container with ID starting with 0d8a364542cd5799bc8ae15a8ef9827aad96c68962f0aacfd63ee7e71b641f76 not found: ID does not exist" containerID="0d8a364542cd5799bc8ae15a8ef9827aad96c68962f0aacfd63ee7e71b641f76" Oct 30 01:41:00.165700 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:00.165653 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d8a364542cd5799bc8ae15a8ef9827aad96c68962f0aacfd63ee7e71b641f76"} err="failed to get container status \"0d8a364542cd5799bc8ae15a8ef9827aad96c68962f0aacfd63ee7e71b641f76\": rpc error: code = NotFound desc = could not find container \"0d8a364542cd5799bc8ae15a8ef9827aad96c68962f0aacfd63ee7e71b641f76\": container with ID starting with 0d8a364542cd5799bc8ae15a8ef9827aad96c68962f0aacfd63ee7e71b641f76 not found: ID does not exist" Oct 30 01:41:00.229711 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:00.229641 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-19a5f1-56c7d78bc7-4bp7l"] Oct 30 01:41:00.256556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:00.253114 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-19a5f1-56c7d78bc7-4bp7l"] Oct 30 01:41:00.547714 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:00.547682 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2eebf2d-067e-4524-a8d2-eb97208c89e4" path="/var/lib/kubelet/pods/a2eebf2d-067e-4524-a8d2-eb97208c89e4/volumes" Oct 30 01:41:00.985796 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:00.985713 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-6b3dbd/ipp-route-6b3dbd-kserve-894c568b9-d6fnn" Oct 30 01:41:00.998047 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:00.998018 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-6b3dbd/ipp-route-6b3dbd-kserve-894c568b9-d6fnn" Oct 30 01:41:01.170811 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:01.170778 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-6b3dbd/ipp-route-6b3dbd-kserve-894c568b9-d6fnn" Oct 30 01:41:02.080609 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:02.080574 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db534-77cccfc547-jh5tl" Oct 30 01:41:06.195591 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:06.195481 6221 generic.go:356] "Generic (PLEG): container finished" podID="cda4f86d-d23b-4da5-824e-cacea3b08ab7" containerID="26843365aab8f7669d8a549e01e9df0149f0c9cccb5e58dae07bc8a3061e6898" exitCode=0 Oct 30 01:41:06.196096 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:06.195600 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" event={"ID":"cda4f86d-d23b-4da5-824e-cacea3b08ab7","Type":"ContainerDied","Data":"26843365aab8f7669d8a549e01e9df0149f0c9cccb5e58dae07bc8a3061e6898"} Oct 30 01:41:06.196096 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:06.196036 6221 scope.go:122] "RemoveContainer" containerID="26843365aab8f7669d8a549e01e9df0149f0c9cccb5e58dae07bc8a3061e6898" Oct 30 01:41:06.197721 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:06.197694 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:41:06.197808 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:06.197764 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:41:06.197808 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:06.197795 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:41:07.203721 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:07.203689 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" event={"ID":"cda4f86d-d23b-4da5-824e-cacea3b08ab7","Type":"ContainerStarted","Data":"03e4821dd8e93556f89cf8e286a1b31e0b1e9b96178d11298deeca6d5c88cd46"} Oct 30 01:41:07.246781 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:07.246738 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9"] Oct 30 01:41:07.247247 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:07.247199 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" podUID="43a3609a-8ccb-4149-82ec-98834c8f95a3" containerName="istio-proxy" containerID="cri-o://222cd8b4e05700e20e788a1057953a84cdf53808bfa380db63a30d9445586484" gracePeriod=30 Oct 30 01:41:07.286484 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:07.286459 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" Oct 30 01:41:07.287466 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:07.287443 6221 patch_prober.go:29] interesting pod/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 01:41:07.287580 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:07.287487 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" podUID="cda4f86d-d23b-4da5-824e-cacea3b08ab7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 01:41:08.286654 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:08.286619 6221 patch_prober.go:29] interesting pod/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 01:41:08.287128 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:08.286674 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" podUID="cda4f86d-d23b-4da5-824e-cacea3b08ab7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 01:41:08.480870 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:08.480838 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" Oct 30 01:41:09.025682 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:09.025628 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f01cbc-77449896b-mzbkb"] Oct 30 01:41:09.027080 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:09.026967 6221 cpu_manager.go:414] "containerMap: removing container" podUID="a2eebf2d-067e-4524-a8d2-eb97208c89e4" containerName="maas-api" Oct 30 01:41:09.027269 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:09.027251 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2eebf2d-067e-4524-a8d2-eb97208c89e4" containerName="maas-api" Oct 30 01:41:09.027605 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:09.027587 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2eebf2d-067e-4524-a8d2-eb97208c89e4" containerName="maas-api" Oct 30 01:41:09.050880 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:09.050839 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f01cbc-77449896b-mzbkb" Oct 30 01:41:09.054228 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:09.054195 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f01cbc-77449896b-mzbkb"] Oct 30 01:41:09.067475 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:09.067445 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f01cbc\"" Oct 30 01:41:09.081899 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:09.081870 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f01cbc-dockercfg-wvvf5\"" Oct 30 01:41:09.156514 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:09.156484 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmrqf\" (UniqueName: \"kubernetes.io/projected/74c960b2-ff58-455b-8eb4-43cc4171e32c-kube-api-access-bmrqf\") pod \"payload-processing-e2e-ipp-b-f01cbc-77449896b-mzbkb\" (UID: \"74c960b2-ff58-455b-8eb4-43cc4171e32c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f01cbc-77449896b-mzbkb" Oct 30 01:41:09.156731 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:09.156678 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74c960b2-ff58-455b-8eb4-43cc4171e32c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f01cbc-77449896b-mzbkb\" (UID: \"74c960b2-ff58-455b-8eb4-43cc4171e32c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f01cbc-77449896b-mzbkb" Oct 30 01:41:09.202374 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:09.202343 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f01cbc-7f6fdd8bb7-pkht9"] Oct 30 01:41:09.219815 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:09.219786 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f01cbc-7f6fdd8bb7-pkht9" Oct 30 01:41:09.236562 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:09.236515 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f01cbc-7f6fdd8bb7-pkht9"] Oct 30 01:41:09.257991 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:09.257752 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74c960b2-ff58-455b-8eb4-43cc4171e32c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f01cbc-77449896b-mzbkb\" (UID: \"74c960b2-ff58-455b-8eb4-43cc4171e32c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f01cbc-77449896b-mzbkb" Oct 30 01:41:09.258152 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:09.257987 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmrqf\" (UniqueName: \"kubernetes.io/projected/74c960b2-ff58-455b-8eb4-43cc4171e32c-kube-api-access-bmrqf\") pod \"payload-processing-e2e-ipp-b-f01cbc-77449896b-mzbkb\" (UID: \"74c960b2-ff58-455b-8eb4-43cc4171e32c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f01cbc-77449896b-mzbkb" Oct 30 01:41:09.258152 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:09.258040 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqj7c\" (UniqueName: \"kubernetes.io/projected/8f2b1f15-868f-4126-bf67-b65a2e5b50a7-kube-api-access-dqj7c\") pod \"payload-pre-processing-e2e-ipp-b-f01cbc-7f6fdd8bb7-pkht9\" (UID: \"8f2b1f15-868f-4126-bf67-b65a2e5b50a7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f01cbc-7f6fdd8bb7-pkht9" Oct 30 01:41:09.258596 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:09.258564 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f2b1f15-868f-4126-bf67-b65a2e5b50a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f01cbc-7f6fdd8bb7-pkht9\" (UID: \"8f2b1f15-868f-4126-bf67-b65a2e5b50a7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f01cbc-7f6fdd8bb7-pkht9" Oct 30 01:41:09.258722 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:09.258635 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74c960b2-ff58-455b-8eb4-43cc4171e32c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f01cbc-77449896b-mzbkb\" (UID: \"74c960b2-ff58-455b-8eb4-43cc4171e32c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f01cbc-77449896b-mzbkb" Oct 30 01:41:09.278712 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:09.278661 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmrqf\" (UniqueName: \"kubernetes.io/projected/74c960b2-ff58-455b-8eb4-43cc4171e32c-kube-api-access-bmrqf\") pod \"payload-processing-e2e-ipp-b-f01cbc-77449896b-mzbkb\" (UID: \"74c960b2-ff58-455b-8eb4-43cc4171e32c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f01cbc-77449896b-mzbkb" Oct 30 01:41:09.287165 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:09.287141 6221 patch_prober.go:29] interesting pod/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 01:41:09.287456 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:09.287187 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" podUID="cda4f86d-d23b-4da5-824e-cacea3b08ab7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 01:41:09.360006 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:09.359962 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqj7c\" (UniqueName: \"kubernetes.io/projected/8f2b1f15-868f-4126-bf67-b65a2e5b50a7-kube-api-access-dqj7c\") pod \"payload-pre-processing-e2e-ipp-b-f01cbc-7f6fdd8bb7-pkht9\" (UID: \"8f2b1f15-868f-4126-bf67-b65a2e5b50a7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f01cbc-7f6fdd8bb7-pkht9" Oct 30 01:41:09.360195 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:09.360059 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f2b1f15-868f-4126-bf67-b65a2e5b50a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f01cbc-7f6fdd8bb7-pkht9\" (UID: \"8f2b1f15-868f-4126-bf67-b65a2e5b50a7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f01cbc-7f6fdd8bb7-pkht9" Oct 30 01:41:09.360799 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:09.360779 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f2b1f15-868f-4126-bf67-b65a2e5b50a7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f01cbc-7f6fdd8bb7-pkht9\" (UID: \"8f2b1f15-868f-4126-bf67-b65a2e5b50a7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f01cbc-7f6fdd8bb7-pkht9" Oct 30 01:41:09.379711 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:09.379678 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqj7c\" (UniqueName: \"kubernetes.io/projected/8f2b1f15-868f-4126-bf67-b65a2e5b50a7-kube-api-access-dqj7c\") pod \"payload-pre-processing-e2e-ipp-b-f01cbc-7f6fdd8bb7-pkht9\" (UID: \"8f2b1f15-868f-4126-bf67-b65a2e5b50a7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f01cbc-7f6fdd8bb7-pkht9" Oct 30 01:41:09.387564 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:09.387543 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f01cbc-77449896b-mzbkb" Oct 30 01:41:09.550518 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:09.550465 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f01cbc-7f6fdd8bb7-pkht9" Oct 30 01:41:09.736237 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:09.736111 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f01cbc-77449896b-mzbkb"] Oct 30 01:41:09.739436 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:41:09.739404 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74c960b2_ff58_455b_8eb4_43cc4171e32c.slice/crio-d63c7520799ba30f4aca5452bb55134e38da07bec21dcf9587cbcb2c9d1dbfe9 WatchSource:0}: Error finding container d63c7520799ba30f4aca5452bb55134e38da07bec21dcf9587cbcb2c9d1dbfe9: Status 404 returned error can't find the container with id d63c7520799ba30f4aca5452bb55134e38da07bec21dcf9587cbcb2c9d1dbfe9 Oct 30 01:41:09.900757 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:09.900728 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f01cbc-7f6fdd8bb7-pkht9"] Oct 30 01:41:09.905435 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:41:09.905228 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f2b1f15_868f_4126_bf67_b65a2e5b50a7.slice/crio-25b7e26051df30ea072ee4d64eb041ea1b4003723456673c435b98ec21b85f76 WatchSource:0}: Error finding container 25b7e26051df30ea072ee4d64eb041ea1b4003723456673c435b98ec21b85f76: Status 404 returned error can't find the container with id 25b7e26051df30ea072ee4d64eb041ea1b4003723456673c435b98ec21b85f76 Oct 30 01:41:10.247691 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:10.247598 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f01cbc-7f6fdd8bb7-pkht9" event={"ID":"8f2b1f15-868f-4126-bf67-b65a2e5b50a7","Type":"ContainerStarted","Data":"5b1475a1959ebbf2478c733b97d39e4c65275bd63ebbd71fb271e288dee05961"} Oct 30 01:41:10.247691 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:10.247663 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f01cbc-7f6fdd8bb7-pkht9" event={"ID":"8f2b1f15-868f-4126-bf67-b65a2e5b50a7","Type":"ContainerStarted","Data":"25b7e26051df30ea072ee4d64eb041ea1b4003723456673c435b98ec21b85f76"} Oct 30 01:41:10.247924 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:10.247693 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f01cbc-7f6fdd8bb7-pkht9" Oct 30 01:41:10.249261 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:10.249231 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f01cbc-77449896b-mzbkb" event={"ID":"74c960b2-ff58-455b-8eb4-43cc4171e32c","Type":"ContainerStarted","Data":"40e42cc878a25ff472659ca140970cd03db607fc5f7620e22d7309d8130d5758"} Oct 30 01:41:10.249382 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:10.249289 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f01cbc-77449896b-mzbkb" event={"ID":"74c960b2-ff58-455b-8eb4-43cc4171e32c","Type":"ContainerStarted","Data":"d63c7520799ba30f4aca5452bb55134e38da07bec21dcf9587cbcb2c9d1dbfe9"} Oct 30 01:41:10.249382 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:10.249317 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f01cbc-77449896b-mzbkb" Oct 30 01:41:10.286963 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:10.286927 6221 patch_prober.go:29] interesting pod/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 01:41:10.287141 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:10.287010 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" podUID="cda4f86d-d23b-4da5-824e-cacea3b08ab7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 01:41:10.289948 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:10.289901 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f01cbc-7f6fdd8bb7-pkht9" podStartSLOduration=1.289889052 podStartE2EDuration="1.289889052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:41:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:41:10.286834949 +0000 UTC m=+1393.865122286" watchObservedRunningTime="2026-10-30 01:41:10.289889052 +0000 UTC m=+1393.868176389" Oct 30 01:41:11.316377 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:11.316334 6221 patch_prober.go:29] interesting pod/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 01:41:11.317154 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:11.317120 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" podUID="cda4f86d-d23b-4da5-824e-cacea3b08ab7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 01:41:12.288335 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:12.288304 6221 patch_prober.go:29] interesting pod/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body= Oct 30 01:41:12.288461 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:12.288357 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" podUID="cda4f86d-d23b-4da5-824e-cacea3b08ab7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" Oct 30 01:41:12.371196 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:12.371168 6221 generic.go:356] "Generic (PLEG): container finished" podID="43a3609a-8ccb-4149-82ec-98834c8f95a3" containerID="222cd8b4e05700e20e788a1057953a84cdf53808bfa380db63a30d9445586484" exitCode=0 Oct 30 01:41:12.371636 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:12.371240 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" event={"ID":"43a3609a-8ccb-4149-82ec-98834c8f95a3","Type":"ContainerDied","Data":"222cd8b4e05700e20e788a1057953a84cdf53808bfa380db63a30d9445586484"} Oct 30 01:41:12.371636 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:12.371347 6221 scope.go:122] "RemoveContainer" containerID="3fd81a3eef3c036d1ffa2979aeba8042424968be064164f5de527228f05d70b2" Oct 30 01:41:12.591623 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:12.591592 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" Oct 30 01:41:12.635655 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:12.635580 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f01cbc-77449896b-mzbkb" podStartSLOduration=4.635562723 podStartE2EDuration="4.635562723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:41:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:41:10.331837109 +0000 UTC m=+1393.910124445" watchObservedRunningTime="2026-10-30 01:41:12.635562723 +0000 UTC m=+1396.213850079" Oct 30 01:41:12.661917 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:12.661878 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43a3609a-8ccb-4149-82ec-98834c8f95a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43a3609a-8ccb-4149-82ec-98834c8f95a3-istio-data\") pod \"43a3609a-8ccb-4149-82ec-98834c8f95a3\" (UID: \"43a3609a-8ccb-4149-82ec-98834c8f95a3\") " Oct 30 01:41:12.662121 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:12.661995 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43a3609a-8ccb-4149-82ec-98834c8f95a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a3609a-8ccb-4149-82ec-98834c8f95a3-workload-socket\") pod \"43a3609a-8ccb-4149-82ec-98834c8f95a3\" (UID: \"43a3609a-8ccb-4149-82ec-98834c8f95a3\") " Oct 30 01:41:12.662121 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:12.662041 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43a3609a-8ccb-4149-82ec-98834c8f95a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43a3609a-8ccb-4149-82ec-98834c8f95a3-workload-certs\") pod \"43a3609a-8ccb-4149-82ec-98834c8f95a3\" (UID: \"43a3609a-8ccb-4149-82ec-98834c8f95a3\") " Oct 30 01:41:12.662260 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:12.662132 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43a3609a-8ccb-4149-82ec-98834c8f95a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43a3609a-8ccb-4149-82ec-98834c8f95a3-istio-podinfo\") pod \"43a3609a-8ccb-4149-82ec-98834c8f95a3\" (UID: \"43a3609a-8ccb-4149-82ec-98834c8f95a3\") " Oct 30 01:41:12.662260 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:12.662166 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43a3609a-8ccb-4149-82ec-98834c8f95a3-istio-token\" (UniqueName: \"kubernetes.io/projected/43a3609a-8ccb-4149-82ec-98834c8f95a3-istio-token\") pod \"43a3609a-8ccb-4149-82ec-98834c8f95a3\" (UID: \"43a3609a-8ccb-4149-82ec-98834c8f95a3\") " Oct 30 01:41:12.662775 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:12.662739 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a3609a-8ccb-4149-82ec-98834c8f95a3-workload-certs" pod "43a3609a-8ccb-4149-82ec-98834c8f95a3" (UID: "43a3609a-8ccb-4149-82ec-98834c8f95a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:41:12.663019 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:12.662993 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a3609a-8ccb-4149-82ec-98834c8f95a3-workload-socket" pod "43a3609a-8ccb-4149-82ec-98834c8f95a3" (UID: "43a3609a-8ccb-4149-82ec-98834c8f95a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:41:12.663106 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:12.663070 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a3609a-8ccb-4149-82ec-98834c8f95a3-istio-data" pod "43a3609a-8ccb-4149-82ec-98834c8f95a3" (UID: "43a3609a-8ccb-4149-82ec-98834c8f95a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:41:12.664173 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:12.664149 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43a3609a-8ccb-4149-82ec-98834c8f95a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a3609a-8ccb-4149-82ec-98834c8f95a3-credential-socket\") pod \"43a3609a-8ccb-4149-82ec-98834c8f95a3\" (UID: \"43a3609a-8ccb-4149-82ec-98834c8f95a3\") " Oct 30 01:41:12.664380 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:12.664362 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43a3609a-8ccb-4149-82ec-98834c8f95a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43a3609a-8ccb-4149-82ec-98834c8f95a3-istiod-ca-cert\") pod \"43a3609a-8ccb-4149-82ec-98834c8f95a3\" (UID: \"43a3609a-8ccb-4149-82ec-98834c8f95a3\") " Oct 30 01:41:12.664567 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:12.664513 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43a3609a-8ccb-4149-82ec-98834c8f95a3-kube-api-access-j8zrz\" (UniqueName: \"kubernetes.io/projected/43a3609a-8ccb-4149-82ec-98834c8f95a3-kube-api-access-j8zrz\") pod \"43a3609a-8ccb-4149-82ec-98834c8f95a3\" (UID: \"43a3609a-8ccb-4149-82ec-98834c8f95a3\") " Oct 30 01:41:12.664750 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:12.664412 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a3609a-8ccb-4149-82ec-98834c8f95a3-credential-socket" pod "43a3609a-8ccb-4149-82ec-98834c8f95a3" (UID: "43a3609a-8ccb-4149-82ec-98834c8f95a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:41:12.664836 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:12.664724 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43a3609a-8ccb-4149-82ec-98834c8f95a3-istiod-ca-cert" pod "43a3609a-8ccb-4149-82ec-98834c8f95a3" (UID: "43a3609a-8ccb-4149-82ec-98834c8f95a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:41:12.665029 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:12.665010 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43a3609a-8ccb-4149-82ec-98834c8f95a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43a3609a-8ccb-4149-82ec-98834c8f95a3-istio-envoy\") pod \"43a3609a-8ccb-4149-82ec-98834c8f95a3\" (UID: \"43a3609a-8ccb-4149-82ec-98834c8f95a3\") " Oct 30 01:41:12.665883 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:12.665861 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43a3609a-8ccb-4149-82ec-98834c8f95a3-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:41:12.666009 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:12.665996 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a3609a-8ccb-4149-82ec-98834c8f95a3-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:41:12.666100 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:12.666087 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43a3609a-8ccb-4149-82ec-98834c8f95a3-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:41:12.666184 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:12.666173 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43a3609a-8ccb-4149-82ec-98834c8f95a3-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:41:12.666258 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:12.666248 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43a3609a-8ccb-4149-82ec-98834c8f95a3-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:41:12.666426 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:12.666405 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43a3609a-8ccb-4149-82ec-98834c8f95a3-istio-podinfo" pod "43a3609a-8ccb-4149-82ec-98834c8f95a3" (UID: "43a3609a-8ccb-4149-82ec-98834c8f95a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:41:12.666506 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:12.666467 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43a3609a-8ccb-4149-82ec-98834c8f95a3-istio-token" pod "43a3609a-8ccb-4149-82ec-98834c8f95a3" (UID: "43a3609a-8ccb-4149-82ec-98834c8f95a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:41:12.671106 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:12.671082 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a3609a-8ccb-4149-82ec-98834c8f95a3-istio-envoy" pod "43a3609a-8ccb-4149-82ec-98834c8f95a3" (UID: "43a3609a-8ccb-4149-82ec-98834c8f95a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:41:12.671569 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:12.671547 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43a3609a-8ccb-4149-82ec-98834c8f95a3-kube-api-access-j8zrz" pod "43a3609a-8ccb-4149-82ec-98834c8f95a3" (UID: "43a3609a-8ccb-4149-82ec-98834c8f95a3"). InnerVolumeSpecName "kube-api-access-j8zrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:41:12.767845 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:12.767811 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8zrz\" (UniqueName: \"kubernetes.io/projected/43a3609a-8ccb-4149-82ec-98834c8f95a3-kube-api-access-j8zrz\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:41:12.767845 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:12.767843 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43a3609a-8ccb-4149-82ec-98834c8f95a3-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:41:12.768075 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:12.767857 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43a3609a-8ccb-4149-82ec-98834c8f95a3-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:41:12.768075 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:12.767870 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43a3609a-8ccb-4149-82ec-98834c8f95a3-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:41:13.291593 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:13.291563 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" Oct 30 01:41:13.292489 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:13.292469 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" Oct 30 01:41:13.379221 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:13.379184 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" Oct 30 01:41:13.379730 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:13.379226 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9" event={"ID":"43a3609a-8ccb-4149-82ec-98834c8f95a3","Type":"ContainerDied","Data":"b22888479b59abea944b3bf82b3efc26c514cace76a03750c2d58e18ce630875"} Oct 30 01:41:13.379730 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:13.379290 6221 scope.go:122] "RemoveContainer" containerID="222cd8b4e05700e20e788a1057953a84cdf53808bfa380db63a30d9445586484" Oct 30 01:41:13.430884 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:13.430850 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9"] Oct 30 01:41:13.434621 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:13.434594 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-19a5f1-openshift-default-6cf4d9c44f-mqrq9"] Oct 30 01:41:14.548659 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:14.548626 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43a3609a-8ccb-4149-82ec-98834c8f95a3" path="/var/lib/kubelet/pods/43a3609a-8ccb-4149-82ec-98834c8f95a3/volumes" Oct 30 01:41:16.275500 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:16.275465 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2ac18f-6c4f9cfb86-2gt8n"] Oct 30 01:41:21.276060 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:21.275998 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-2ac18f-6c4f9cfb86-2gt8n" podUID="3db79759-6455-4d30-8d81-042eaf326058" containerName="payload-processing" containerID="cri-o://18743d27de0d1bfd957bc31cb1107402f124345926e8705fd8247fa9e179b99f" gracePeriod=25 Oct 30 01:41:21.389130 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:21.389102 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f01cbc-7f6fdd8bb7-pkht9" Oct 30 01:41:21.389305 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:21.389277 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f01cbc-77449896b-mzbkb" Oct 30 01:41:21.436635 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:21.436593 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2ac18f-6c4f9cfb86-2gt8n" event={"ID":"3db79759-6455-4d30-8d81-042eaf326058","Type":"ContainerDied","Data":"18743d27de0d1bfd957bc31cb1107402f124345926e8705fd8247fa9e179b99f"} Oct 30 01:41:21.436816 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:21.436523 6221 generic.go:356] "Generic (PLEG): container finished" podID="3db79759-6455-4d30-8d81-042eaf326058" containerID="18743d27de0d1bfd957bc31cb1107402f124345926e8705fd8247fa9e179b99f" exitCode=0 Oct 30 01:41:21.507514 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:41:21.507471 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:41:21.585351 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:21.585327 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2ac18f-6c4f9cfb86-2gt8n" Oct 30 01:41:21.676353 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:21.674038 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3db79759-6455-4d30-8d81-042eaf326058-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3db79759-6455-4d30-8d81-042eaf326058-plugins-config-volume\") pod \"3db79759-6455-4d30-8d81-042eaf326058\" (UID: \"3db79759-6455-4d30-8d81-042eaf326058\") " Oct 30 01:41:21.676353 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:21.674226 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3db79759-6455-4d30-8d81-042eaf326058-kube-api-access-g48sr\" (UniqueName: \"kubernetes.io/projected/3db79759-6455-4d30-8d81-042eaf326058-kube-api-access-g48sr\") pod \"3db79759-6455-4d30-8d81-042eaf326058\" (UID: \"3db79759-6455-4d30-8d81-042eaf326058\") " Oct 30 01:41:21.679837 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:21.679787 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3db79759-6455-4d30-8d81-042eaf326058-plugins-config-volume" pod "3db79759-6455-4d30-8d81-042eaf326058" (UID: "3db79759-6455-4d30-8d81-042eaf326058"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:41:21.684829 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:21.684800 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3db79759-6455-4d30-8d81-042eaf326058-kube-api-access-g48sr" pod "3db79759-6455-4d30-8d81-042eaf326058" (UID: "3db79759-6455-4d30-8d81-042eaf326058"). InnerVolumeSpecName "kube-api-access-g48sr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:41:21.812656 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:21.812568 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g48sr\" (UniqueName: \"kubernetes.io/projected/3db79759-6455-4d30-8d81-042eaf326058-kube-api-access-g48sr\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:41:21.812873 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:21.812857 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3db79759-6455-4d30-8d81-042eaf326058-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:41:22.443817 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:22.443784 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2ac18f-6c4f9cfb86-2gt8n" Oct 30 01:41:22.444326 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:22.443789 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2ac18f-6c4f9cfb86-2gt8n" event={"ID":"3db79759-6455-4d30-8d81-042eaf326058","Type":"ContainerDied","Data":"f925bff31e8aaad1a6dc5e4f82ca290790aa8f0c3ebc86efbb130a3e0220fdca"} Oct 30 01:41:22.444326 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:22.443923 6221 scope.go:122] "RemoveContainer" containerID="18743d27de0d1bfd957bc31cb1107402f124345926e8705fd8247fa9e179b99f" Oct 30 01:41:22.502662 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:22.502631 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2ac18f-6c4f9cfb86-2gt8n"] Oct 30 01:41:22.514903 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:22.514873 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2ac18f-6c4f9cfb86-2gt8n"] Oct 30 01:41:22.547897 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:22.547866 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3db79759-6455-4d30-8d81-042eaf326058" path="/var/lib/kubelet/pods/3db79759-6455-4d30-8d81-042eaf326058/volumes" Oct 30 01:41:35.356631 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:35.356599 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:41:36.366948 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:36.366914 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3fde1c-6c6f84fb95-4d8dt"] Oct 30 01:41:36.368248 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:36.368224 6221 cpu_manager.go:414] "containerMap: removing container" podUID="43a3609a-8ccb-4149-82ec-98834c8f95a3" containerName="istio-proxy" Oct 30 01:41:36.368377 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:36.368365 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43a3609a-8ccb-4149-82ec-98834c8f95a3" containerName="istio-proxy" Oct 30 01:41:36.368497 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:36.368488 6221 cpu_manager.go:414] "containerMap: removing container" podUID="43a3609a-8ccb-4149-82ec-98834c8f95a3" containerName="istio-proxy" Oct 30 01:41:36.368579 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:36.368569 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43a3609a-8ccb-4149-82ec-98834c8f95a3" containerName="istio-proxy" Oct 30 01:41:36.368654 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:36.368645 6221 cpu_manager.go:414] "containerMap: removing container" podUID="3db79759-6455-4d30-8d81-042eaf326058" containerName="payload-processing" Oct 30 01:41:36.368716 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:36.368707 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3db79759-6455-4d30-8d81-042eaf326058" containerName="payload-processing" Oct 30 01:41:36.368955 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:36.368943 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="43a3609a-8ccb-4149-82ec-98834c8f95a3" containerName="istio-proxy" Oct 30 01:41:36.369033 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:36.369025 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="3db79759-6455-4d30-8d81-042eaf326058" containerName="payload-processing" Oct 30 01:41:36.369112 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:36.369103 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="43a3609a-8ccb-4149-82ec-98834c8f95a3" containerName="istio-proxy" Oct 30 01:41:36.389053 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:36.389026 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3fde1c-6c6f84fb95-4d8dt" Oct 30 01:41:36.389701 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:36.389674 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3fde1c-6c6f84fb95-4d8dt"] Oct 30 01:41:36.449557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:36.429631 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-3fde1c\"" Oct 30 01:41:36.449557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:36.429771 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-3fde1c-dockercfg-nz8ch\"" Oct 30 01:41:36.618488 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:36.618395 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klfbv\" (UniqueName: \"kubernetes.io/projected/f0f39f0a-f1f1-41aa-a9d9-4df2e419ad59-kube-api-access-klfbv\") pod \"payload-pre-processing-e2e-praxis-a-3fde1c-6c6f84fb95-4d8dt\" (UID: \"f0f39f0a-f1f1-41aa-a9d9-4df2e419ad59\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3fde1c-6c6f84fb95-4d8dt" Oct 30 01:41:36.618844 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:36.618817 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0f39f0a-f1f1-41aa-a9d9-4df2e419ad59-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-3fde1c-6c6f84fb95-4d8dt\" (UID: \"f0f39f0a-f1f1-41aa-a9d9-4df2e419ad59\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3fde1c-6c6f84fb95-4d8dt" Oct 30 01:41:36.720645 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:36.720608 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klfbv\" (UniqueName: \"kubernetes.io/projected/f0f39f0a-f1f1-41aa-a9d9-4df2e419ad59-kube-api-access-klfbv\") pod \"payload-pre-processing-e2e-praxis-a-3fde1c-6c6f84fb95-4d8dt\" (UID: \"f0f39f0a-f1f1-41aa-a9d9-4df2e419ad59\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3fde1c-6c6f84fb95-4d8dt" Oct 30 01:41:36.720829 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:36.720684 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0f39f0a-f1f1-41aa-a9d9-4df2e419ad59-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-3fde1c-6c6f84fb95-4d8dt\" (UID: \"f0f39f0a-f1f1-41aa-a9d9-4df2e419ad59\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3fde1c-6c6f84fb95-4d8dt" Oct 30 01:41:36.721399 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:36.721377 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0f39f0a-f1f1-41aa-a9d9-4df2e419ad59-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-3fde1c-6c6f84fb95-4d8dt\" (UID: \"f0f39f0a-f1f1-41aa-a9d9-4df2e419ad59\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3fde1c-6c6f84fb95-4d8dt" Oct 30 01:41:36.740570 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:36.740517 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klfbv\" (UniqueName: \"kubernetes.io/projected/f0f39f0a-f1f1-41aa-a9d9-4df2e419ad59-kube-api-access-klfbv\") pod \"payload-pre-processing-e2e-praxis-a-3fde1c-6c6f84fb95-4d8dt\" (UID: \"f0f39f0a-f1f1-41aa-a9d9-4df2e419ad59\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3fde1c-6c6f84fb95-4d8dt" Oct 30 01:41:37.007926 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:37.007845 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3fde1c-6c6f84fb95-4d8dt" Oct 30 01:41:37.315025 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:37.310569 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3fde1c-6c6f84fb95-4d8dt"] Oct 30 01:41:37.439675 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:41:37.439642 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0f39f0a_f1f1_41aa_a9d9_4df2e419ad59.slice/crio-988acac4ae18d768679ebaa1644fa1c6335f279c2344a9e658483c59cb3ca9ad WatchSource:0}: Error finding container 988acac4ae18d768679ebaa1644fa1c6335f279c2344a9e658483c59cb3ca9ad: Status 404 returned error can't find the container with id 988acac4ae18d768679ebaa1644fa1c6335f279c2344a9e658483c59cb3ca9ad Oct 30 01:41:37.551970 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:37.551450 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2ac18f-9db867f76-qwr6n"] Oct 30 01:41:37.569224 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:37.569161 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2ac18f-9db867f76-qwr6n" Oct 30 01:41:37.580339 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:37.580312 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2ac18f-9db867f76-qwr6n"] Oct 30 01:41:37.589571 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:37.589165 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2ac18f-dockercfg-mfq7h\"" Oct 30 01:41:37.589571 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:37.589252 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2ac18f\"" Oct 30 01:41:37.684397 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:37.684361 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3fde1c-6c6f84fb95-4d8dt" event={"ID":"f0f39f0a-f1f1-41aa-a9d9-4df2e419ad59","Type":"ContainerStarted","Data":"ea7b368d3ce0389a108e8819cf68a9660565f7ebad1b54ce78062e44e2185bd9"} Oct 30 01:41:37.684544 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:37.684420 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3fde1c-6c6f84fb95-4d8dt" event={"ID":"f0f39f0a-f1f1-41aa-a9d9-4df2e419ad59","Type":"ContainerStarted","Data":"988acac4ae18d768679ebaa1644fa1c6335f279c2344a9e658483c59cb3ca9ad"} Oct 30 01:41:37.684611 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:37.684548 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3fde1c-6c6f84fb95-4d8dt" Oct 30 01:41:37.708670 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:37.708409 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f526236-9252-471e-97da-3318a14dd75a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2ac18f-9db867f76-qwr6n\" (UID: \"6f526236-9252-471e-97da-3318a14dd75a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2ac18f-9db867f76-qwr6n" Oct 30 01:41:37.708670 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:37.708545 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtc4s\" (UniqueName: \"kubernetes.io/projected/6f526236-9252-471e-97da-3318a14dd75a-kube-api-access-dtc4s\") pod \"payload-processing-e2e-ipp-a-2ac18f-9db867f76-qwr6n\" (UID: \"6f526236-9252-471e-97da-3318a14dd75a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2ac18f-9db867f76-qwr6n" Oct 30 01:41:37.724640 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:37.724588 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3fde1c-6c6f84fb95-4d8dt" podStartSLOduration=1.724573891 podStartE2EDuration="1.724573891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:41:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:41:37.721615341 +0000 UTC m=+1421.299902677" watchObservedRunningTime="2026-10-30 01:41:37.724573891 +0000 UTC m=+1421.302861228" Oct 30 01:41:37.764800 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:37.764766 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7bdcc-57cf44cdf5-x2vkk"] Oct 30 01:41:37.764969 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:37.764945 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7bdcc-57cf44cdf5-x2vkk" podUID="41ea8723-6c14-44b1-8926-ae762a267b6e" containerName="maas-api" containerID="cri-o://6ff3151435cc343516d8dfa50aad4d108ff7809fa875929f179fc4f5515bdfea" gracePeriod=30 Oct 30 01:41:37.810325 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:37.810286 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f526236-9252-471e-97da-3318a14dd75a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2ac18f-9db867f76-qwr6n\" (UID: \"6f526236-9252-471e-97da-3318a14dd75a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2ac18f-9db867f76-qwr6n" Oct 30 01:41:37.810485 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:37.810367 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtc4s\" (UniqueName: \"kubernetes.io/projected/6f526236-9252-471e-97da-3318a14dd75a-kube-api-access-dtc4s\") pod \"payload-processing-e2e-ipp-a-2ac18f-9db867f76-qwr6n\" (UID: \"6f526236-9252-471e-97da-3318a14dd75a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2ac18f-9db867f76-qwr6n" Oct 30 01:41:37.811246 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:37.811217 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f526236-9252-471e-97da-3318a14dd75a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2ac18f-9db867f76-qwr6n\" (UID: \"6f526236-9252-471e-97da-3318a14dd75a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2ac18f-9db867f76-qwr6n" Oct 30 01:41:37.831838 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:37.831761 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtc4s\" (UniqueName: \"kubernetes.io/projected/6f526236-9252-471e-97da-3318a14dd75a-kube-api-access-dtc4s\") pod \"payload-processing-e2e-ipp-a-2ac18f-9db867f76-qwr6n\" (UID: \"6f526236-9252-471e-97da-3318a14dd75a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2ac18f-9db867f76-qwr6n" Oct 30 01:41:37.915786 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:37.915751 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2ac18f-9db867f76-qwr6n" Oct 30 01:41:38.056307 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:38.056272 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7bdcc-57cf44cdf5-x2vkk" Oct 30 01:41:38.215176 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:38.215104 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41ea8723-6c14-44b1-8926-ae762a267b6e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41ea8723-6c14-44b1-8926-ae762a267b6e-maas-api-tls\") pod \"41ea8723-6c14-44b1-8926-ae762a267b6e\" (UID: \"41ea8723-6c14-44b1-8926-ae762a267b6e\") " Oct 30 01:41:38.215176 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:38.215145 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41ea8723-6c14-44b1-8926-ae762a267b6e-kube-api-access-gfnrv\" (UniqueName: \"kubernetes.io/projected/41ea8723-6c14-44b1-8926-ae762a267b6e-kube-api-access-gfnrv\") pod \"41ea8723-6c14-44b1-8926-ae762a267b6e\" (UID: \"41ea8723-6c14-44b1-8926-ae762a267b6e\") " Oct 30 01:41:38.215401 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:38.215180 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41ea8723-6c14-44b1-8926-ae762a267b6e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/41ea8723-6c14-44b1-8926-ae762a267b6e-metrics-certs\") pod \"41ea8723-6c14-44b1-8926-ae762a267b6e\" (UID: \"41ea8723-6c14-44b1-8926-ae762a267b6e\") " Oct 30 01:41:38.215496 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:38.215476 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41ea8723-6c14-44b1-8926-ae762a267b6e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41ea8723-6c14-44b1-8926-ae762a267b6e-openshift-service-ca\") pod \"41ea8723-6c14-44b1-8926-ae762a267b6e\" (UID: \"41ea8723-6c14-44b1-8926-ae762a267b6e\") " Oct 30 01:41:38.215922 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:38.215881 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41ea8723-6c14-44b1-8926-ae762a267b6e-openshift-service-ca" pod "41ea8723-6c14-44b1-8926-ae762a267b6e" (UID: "41ea8723-6c14-44b1-8926-ae762a267b6e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:41:38.216130 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:38.216112 6221 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41ea8723-6c14-44b1-8926-ae762a267b6e-openshift-service-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:41:38.217400 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:38.217376 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41ea8723-6c14-44b1-8926-ae762a267b6e-maas-api-tls" pod "41ea8723-6c14-44b1-8926-ae762a267b6e" (UID: "41ea8723-6c14-44b1-8926-ae762a267b6e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:41:38.217516 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:38.217399 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41ea8723-6c14-44b1-8926-ae762a267b6e-kube-api-access-gfnrv" pod "41ea8723-6c14-44b1-8926-ae762a267b6e" (UID: "41ea8723-6c14-44b1-8926-ae762a267b6e"). InnerVolumeSpecName "kube-api-access-gfnrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:41:38.217614 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:38.217595 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41ea8723-6c14-44b1-8926-ae762a267b6e-metrics-certs" pod "41ea8723-6c14-44b1-8926-ae762a267b6e" (UID: "41ea8723-6c14-44b1-8926-ae762a267b6e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:41:38.285723 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:38.285686 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2ac18f-9db867f76-qwr6n"] Oct 30 01:41:38.318557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:38.318276 6221 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41ea8723-6c14-44b1-8926-ae762a267b6e-maas-api-tls\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:41:38.318557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:38.318319 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfnrv\" (UniqueName: \"kubernetes.io/projected/41ea8723-6c14-44b1-8926-ae762a267b6e-kube-api-access-gfnrv\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:41:38.318557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:38.318332 6221 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41ea8723-6c14-44b1-8926-ae762a267b6e-metrics-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:41:38.693456 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:38.693404 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2ac18f-9db867f76-qwr6n" event={"ID":"6f526236-9252-471e-97da-3318a14dd75a","Type":"ContainerStarted","Data":"b0c8f1183419f137a1bafba5288b1839e455e565cbfe862de4949bce791c4d3f"} Oct 30 01:41:38.693972 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:38.693470 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2ac18f-9db867f76-qwr6n" event={"ID":"6f526236-9252-471e-97da-3318a14dd75a","Type":"ContainerStarted","Data":"f3b9b8b5d344140c8d306f7a44e1877b00d0674486987e135f8ac2aa05bd4617"} Oct 30 01:41:38.693972 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:38.693511 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2ac18f-9db867f76-qwr6n" Oct 30 01:41:38.695282 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:38.695255 6221 generic.go:356] "Generic (PLEG): container finished" podID="41ea8723-6c14-44b1-8926-ae762a267b6e" containerID="6ff3151435cc343516d8dfa50aad4d108ff7809fa875929f179fc4f5515bdfea" exitCode=0 Oct 30 01:41:38.695429 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:38.695284 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7bdcc-57cf44cdf5-x2vkk" event={"ID":"41ea8723-6c14-44b1-8926-ae762a267b6e","Type":"ContainerDied","Data":"6ff3151435cc343516d8dfa50aad4d108ff7809fa875929f179fc4f5515bdfea"} Oct 30 01:41:38.695429 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:38.695302 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7bdcc-57cf44cdf5-x2vkk" Oct 30 01:41:38.695429 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:38.695337 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7bdcc-57cf44cdf5-x2vkk" event={"ID":"41ea8723-6c14-44b1-8926-ae762a267b6e","Type":"ContainerDied","Data":"ed03c563738fbdc838c503c908a9ffeddcf7887d7d7b307e58fe19176f0d2bff"} Oct 30 01:41:38.695429 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:38.695368 6221 scope.go:122] "RemoveContainer" containerID="6ff3151435cc343516d8dfa50aad4d108ff7809fa875929f179fc4f5515bdfea" Oct 30 01:41:38.706187 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:38.706155 6221 scope.go:122] "RemoveContainer" containerID="6ff3151435cc343516d8dfa50aad4d108ff7809fa875929f179fc4f5515bdfea" Oct 30 01:41:38.706482 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:41:38.706463 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ff3151435cc343516d8dfa50aad4d108ff7809fa875929f179fc4f5515bdfea\": container with ID starting with 6ff3151435cc343516d8dfa50aad4d108ff7809fa875929f179fc4f5515bdfea not found: ID does not exist" containerID="6ff3151435cc343516d8dfa50aad4d108ff7809fa875929f179fc4f5515bdfea" Oct 30 01:41:38.706567 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:38.706496 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ff3151435cc343516d8dfa50aad4d108ff7809fa875929f179fc4f5515bdfea"} err="failed to get container status \"6ff3151435cc343516d8dfa50aad4d108ff7809fa875929f179fc4f5515bdfea\": rpc error: code = NotFound desc = could not find container \"6ff3151435cc343516d8dfa50aad4d108ff7809fa875929f179fc4f5515bdfea\": container with ID starting with 6ff3151435cc343516d8dfa50aad4d108ff7809fa875929f179fc4f5515bdfea not found: ID does not exist" Oct 30 01:41:38.732346 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:38.732306 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-2ac18f-9db867f76-qwr6n" podStartSLOduration=1.732294209 podStartE2EDuration="1.732294209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:41:38.731210688 +0000 UTC m=+1422.309498025" watchObservedRunningTime="2026-10-30 01:41:38.732294209 +0000 UTC m=+1422.310581544" Oct 30 01:41:38.775100 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:38.775071 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7bdcc-57cf44cdf5-x2vkk"] Oct 30 01:41:38.779113 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:38.779092 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f7bdcc-57cf44cdf5-x2vkk"] Oct 30 01:41:40.548093 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:40.548057 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41ea8723-6c14-44b1-8926-ae762a267b6e" path="/var/lib/kubelet/pods/41ea8723-6c14-44b1-8926-ae762a267b6e/volumes" Oct 30 01:41:48.698238 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:48.698210 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3fde1c-6c6f84fb95-4d8dt" Oct 30 01:41:49.716554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:41:49.715749 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2ac18f-9db867f76-qwr6n" Oct 30 01:41:51.494217 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:41:51.494174 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:42:03.418859 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:03.418813 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db534-5b994486c7-lb44h"] Oct 30 01:42:03.419337 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:03.419080 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db534-5b994486c7-lb44h" podUID="ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d" containerName="maas-api" containerID="cri-o://4fdb5a5741bedf42d486abf36e40cbab16970c3d5282559ff1857a48e6b043fc" gracePeriod=30 Oct 30 01:42:03.601895 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:03.601842 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0db534-77cccfc547-jh5tl"] Oct 30 01:42:03.603121 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:03.602347 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db534-77cccfc547-jh5tl" podUID="c5271795-2ae6-4a9e-a49c-dc6f30cbf180" containerName="payload-pre-processing" containerID="cri-o://19a4848c6d399476f6d553db5f523a7671c7bdce20db58053e8b5075f9160f42" gracePeriod=30 Oct 30 01:42:03.768595 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:03.768562 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db534-5b994486c7-lb44h" Oct 30 01:42:03.862303 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:03.862264 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d-openshift-service-ca\") pod \"ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d\" (UID: \"ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d\") " Oct 30 01:42:03.862513 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:03.862378 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d-maas-api-tls\") pod \"ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d\" (UID: \"ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d\") " Oct 30 01:42:03.862513 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:03.862457 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d-metrics-certs\") pod \"ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d\" (UID: \"ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d\") " Oct 30 01:42:03.862679 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:03.862544 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d-kube-api-access-d8dth\" (UniqueName: \"kubernetes.io/projected/ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d-kube-api-access-d8dth\") pod \"ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d\" (UID: \"ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d\") " Oct 30 01:42:03.862767 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:03.862731 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d-openshift-service-ca" pod "ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d" (UID: "ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:42:03.863195 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:03.863166 6221 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d-openshift-service-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:42:03.865491 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:03.865462 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d-maas-api-tls" pod "ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d" (UID: "ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:42:03.865596 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:03.865482 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d-metrics-certs" pod "ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d" (UID: "ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:42:03.865596 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:03.865503 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d-kube-api-access-d8dth" pod "ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d" (UID: "ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d"). InnerVolumeSpecName "kube-api-access-d8dth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:42:03.890075 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:03.890045 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db534-77cccfc547-jh5tl" Oct 30 01:42:03.985613 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:03.969988 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8dth\" (UniqueName: \"kubernetes.io/projected/ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d-kube-api-access-d8dth\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:42:03.985613 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:03.970027 6221 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d-maas-api-tls\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:42:03.985613 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:03.970042 6221 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d-metrics-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:42:04.022584 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:04.021877 6221 generic.go:356] "Generic (PLEG): container finished" podID="ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d" containerID="4fdb5a5741bedf42d486abf36e40cbab16970c3d5282559ff1857a48e6b043fc" exitCode=0 Oct 30 01:42:04.022584 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:04.021966 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db534-5b994486c7-lb44h" event={"ID":"ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d","Type":"ContainerDied","Data":"4fdb5a5741bedf42d486abf36e40cbab16970c3d5282559ff1857a48e6b043fc"} Oct 30 01:42:04.022584 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:04.022013 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db534-5b994486c7-lb44h" event={"ID":"ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d","Type":"ContainerDied","Data":"4ddd9b8eead0979a5d84d91ee6c6fdc39555c455e23d74d3b445d949ed04b5f7"} Oct 30 01:42:04.022584 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:04.022041 6221 scope.go:122] "RemoveContainer" containerID="4fdb5a5741bedf42d486abf36e40cbab16970c3d5282559ff1857a48e6b043fc" Oct 30 01:42:04.022584 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:04.022311 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db534-5b994486c7-lb44h" Oct 30 01:42:04.076496 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:04.076375 6221 generic.go:356] "Generic (PLEG): container finished" podID="c5271795-2ae6-4a9e-a49c-dc6f30cbf180" containerID="19a4848c6d399476f6d553db5f523a7671c7bdce20db58053e8b5075f9160f42" exitCode=0 Oct 30 01:42:04.076806 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:04.076788 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db534-77cccfc547-jh5tl" Oct 30 01:42:04.077031 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:04.076750 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db534-77cccfc547-jh5tl" event={"ID":"c5271795-2ae6-4a9e-a49c-dc6f30cbf180","Type":"ContainerDied","Data":"19a4848c6d399476f6d553db5f523a7671c7bdce20db58053e8b5075f9160f42"} Oct 30 01:42:04.077301 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:04.077187 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-0db534-77cccfc547-jh5tl" event={"ID":"c5271795-2ae6-4a9e-a49c-dc6f30cbf180","Type":"ContainerDied","Data":"f5437f18b0cd68a19c22eeb55f89eb519b0fac0dfa24fe085f98c616853b48d0"} Oct 30 01:42:04.078290 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:04.078270 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5271795-2ae6-4a9e-a49c-dc6f30cbf180-kube-api-access-67jk8\" (UniqueName: \"kubernetes.io/projected/c5271795-2ae6-4a9e-a49c-dc6f30cbf180-kube-api-access-67jk8\") pod \"c5271795-2ae6-4a9e-a49c-dc6f30cbf180\" (UID: \"c5271795-2ae6-4a9e-a49c-dc6f30cbf180\") " Oct 30 01:42:04.083233 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:04.083191 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5271795-2ae6-4a9e-a49c-dc6f30cbf180-plugins-config-volume" pod "c5271795-2ae6-4a9e-a49c-dc6f30cbf180" (UID: "c5271795-2ae6-4a9e-a49c-dc6f30cbf180"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:42:04.083418 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:04.083401 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5271795-2ae6-4a9e-a49c-dc6f30cbf180-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5271795-2ae6-4a9e-a49c-dc6f30cbf180-plugins-config-volume\") pod \"c5271795-2ae6-4a9e-a49c-dc6f30cbf180\" (UID: \"c5271795-2ae6-4a9e-a49c-dc6f30cbf180\") " Oct 30 01:42:04.094329 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:04.094300 6221 scope.go:122] "RemoveContainer" containerID="4fdb5a5741bedf42d486abf36e40cbab16970c3d5282559ff1857a48e6b043fc" Oct 30 01:42:04.095265 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:42:04.095175 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fdb5a5741bedf42d486abf36e40cbab16970c3d5282559ff1857a48e6b043fc\": container with ID starting with 4fdb5a5741bedf42d486abf36e40cbab16970c3d5282559ff1857a48e6b043fc not found: ID does not exist" containerID="4fdb5a5741bedf42d486abf36e40cbab16970c3d5282559ff1857a48e6b043fc" Oct 30 01:42:04.095265 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:04.095221 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fdb5a5741bedf42d486abf36e40cbab16970c3d5282559ff1857a48e6b043fc"} err="failed to get container status \"4fdb5a5741bedf42d486abf36e40cbab16970c3d5282559ff1857a48e6b043fc\": rpc error: code = NotFound desc = could not find container \"4fdb5a5741bedf42d486abf36e40cbab16970c3d5282559ff1857a48e6b043fc\": container with ID starting with 4fdb5a5741bedf42d486abf36e40cbab16970c3d5282559ff1857a48e6b043fc not found: ID does not exist" Oct 30 01:42:04.095265 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:04.095245 6221 scope.go:122] "RemoveContainer" containerID="19a4848c6d399476f6d553db5f523a7671c7bdce20db58053e8b5075f9160f42" Oct 30 01:42:04.097457 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:04.097432 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5271795-2ae6-4a9e-a49c-dc6f30cbf180-kube-api-access-67jk8" pod "c5271795-2ae6-4a9e-a49c-dc6f30cbf180" (UID: "c5271795-2ae6-4a9e-a49c-dc6f30cbf180"). InnerVolumeSpecName "kube-api-access-67jk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:42:04.101803 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:04.101760 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67jk8\" (UniqueName: \"kubernetes.io/projected/c5271795-2ae6-4a9e-a49c-dc6f30cbf180-kube-api-access-67jk8\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:42:04.101803 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:04.101799 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5271795-2ae6-4a9e-a49c-dc6f30cbf180-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:42:04.127766 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:04.127442 6221 scope.go:122] "RemoveContainer" containerID="19a4848c6d399476f6d553db5f523a7671c7bdce20db58053e8b5075f9160f42" Oct 30 01:42:04.128062 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:42:04.127972 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19a4848c6d399476f6d553db5f523a7671c7bdce20db58053e8b5075f9160f42\": container with ID starting with 19a4848c6d399476f6d553db5f523a7671c7bdce20db58053e8b5075f9160f42 not found: ID does not exist" containerID="19a4848c6d399476f6d553db5f523a7671c7bdce20db58053e8b5075f9160f42" Oct 30 01:42:04.128062 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:04.128005 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19a4848c6d399476f6d553db5f523a7671c7bdce20db58053e8b5075f9160f42"} err="failed to get container status \"19a4848c6d399476f6d553db5f523a7671c7bdce20db58053e8b5075f9160f42\": rpc error: code = NotFound desc = could not find container \"19a4848c6d399476f6d553db5f523a7671c7bdce20db58053e8b5075f9160f42\": container with ID starting with 19a4848c6d399476f6d553db5f523a7671c7bdce20db58053e8b5075f9160f42 not found: ID does not exist" Oct 30 01:42:04.167482 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:04.167449 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db534-5b994486c7-lb44h"] Oct 30 01:42:04.170913 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:04.170888 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0db534-5b994486c7-lb44h"] Oct 30 01:42:04.464454 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:04.464419 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0db534-77cccfc547-jh5tl"] Oct 30 01:42:04.472021 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:04.471986 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-0db534-77cccfc547-jh5tl"] Oct 30 01:42:04.548030 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:04.547994 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d" path="/var/lib/kubelet/pods/ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d/volumes" Oct 30 01:42:04.548448 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:04.548434 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5271795-2ae6-4a9e-a49c-dc6f30cbf180" path="/var/lib/kubelet/pods/c5271795-2ae6-4a9e-a49c-dc6f30cbf180/volumes" Oct 30 01:42:05.344379 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:05.344343 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:42:19.357407 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:19.357184 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0ae258-56f894cd47-vmwzz"] Oct 30 01:42:19.358131 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:19.358104 6221 cpu_manager.go:414] "containerMap: removing container" podUID="ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d" containerName="maas-api" Oct 30 01:42:19.358273 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:19.358144 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d" containerName="maas-api" Oct 30 01:42:19.358273 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:19.358202 6221 cpu_manager.go:414] "containerMap: removing container" podUID="c5271795-2ae6-4a9e-a49c-dc6f30cbf180" containerName="payload-pre-processing" Oct 30 01:42:19.358273 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:19.358210 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5271795-2ae6-4a9e-a49c-dc6f30cbf180" containerName="payload-pre-processing" Oct 30 01:42:19.358273 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:19.358220 6221 cpu_manager.go:414] "containerMap: removing container" podUID="41ea8723-6c14-44b1-8926-ae762a267b6e" containerName="maas-api" Oct 30 01:42:19.358273 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:19.358227 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41ea8723-6c14-44b1-8926-ae762a267b6e" containerName="maas-api" Oct 30 01:42:19.358565 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:19.358381 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab33e5bd-3b6d-46d9-ab25-3c4a8a03a88d" containerName="maas-api" Oct 30 01:42:19.358565 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:19.358401 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5271795-2ae6-4a9e-a49c-dc6f30cbf180" containerName="payload-pre-processing" Oct 30 01:42:19.358565 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:19.358411 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="41ea8723-6c14-44b1-8926-ae762a267b6e" containerName="maas-api" Oct 30 01:42:19.362366 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:19.362337 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0ae258-56f894cd47-vmwzz" Oct 30 01:42:19.379486 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:19.379453 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0ae258-56f894cd47-vmwzz"] Oct 30 01:42:19.474607 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:19.471167 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-0ae258-dockercfg-x54h6\"" Oct 30 01:42:19.474607 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:19.471390 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-0ae258\"" Oct 30 01:42:19.498556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:19.476000 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fa64c39-aa67-4869-9164-a3edee1c0be8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-0ae258-56f894cd47-vmwzz\" (UID: \"6fa64c39-aa67-4869-9164-a3edee1c0be8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0ae258-56f894cd47-vmwzz" Oct 30 01:42:19.498556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:19.493833 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8bfq\" (UniqueName: \"kubernetes.io/projected/6fa64c39-aa67-4869-9164-a3edee1c0be8-kube-api-access-x8bfq\") pod \"payload-pre-processing-e2e-praxis-a-0ae258-56f894cd47-vmwzz\" (UID: \"6fa64c39-aa67-4869-9164-a3edee1c0be8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0ae258-56f894cd47-vmwzz" Oct 30 01:42:19.601558 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:19.595444 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8bfq\" (UniqueName: \"kubernetes.io/projected/6fa64c39-aa67-4869-9164-a3edee1c0be8-kube-api-access-x8bfq\") pod \"payload-pre-processing-e2e-praxis-a-0ae258-56f894cd47-vmwzz\" (UID: \"6fa64c39-aa67-4869-9164-a3edee1c0be8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0ae258-56f894cd47-vmwzz" Oct 30 01:42:19.628609 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:19.625306 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fa64c39-aa67-4869-9164-a3edee1c0be8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-0ae258-56f894cd47-vmwzz\" (UID: \"6fa64c39-aa67-4869-9164-a3edee1c0be8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0ae258-56f894cd47-vmwzz" Oct 30 01:42:19.628609 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:19.626323 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fa64c39-aa67-4869-9164-a3edee1c0be8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-0ae258-56f894cd47-vmwzz\" (UID: \"6fa64c39-aa67-4869-9164-a3edee1c0be8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0ae258-56f894cd47-vmwzz" Oct 30 01:42:19.668815 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:19.654381 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8bfq\" (UniqueName: \"kubernetes.io/projected/6fa64c39-aa67-4869-9164-a3edee1c0be8-kube-api-access-x8bfq\") pod \"payload-pre-processing-e2e-praxis-a-0ae258-56f894cd47-vmwzz\" (UID: \"6fa64c39-aa67-4869-9164-a3edee1c0be8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0ae258-56f894cd47-vmwzz" Oct 30 01:42:19.689599 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:19.689281 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0ae258-56f894cd47-vmwzz" Oct 30 01:42:19.987899 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:19.987831 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0ae258-56f894cd47-vmwzz"] Oct 30 01:42:19.990778 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:42:19.990746 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fa64c39_aa67_4869_9164_a3edee1c0be8.slice/crio-6898db5b2223b918d171d9e19ab4e392282f25567034432497eec8168a0aa0be WatchSource:0}: Error finding container 6898db5b2223b918d171d9e19ab4e392282f25567034432497eec8168a0aa0be: Status 404 returned error can't find the container with id 6898db5b2223b918d171d9e19ab4e392282f25567034432497eec8168a0aa0be Oct 30 01:42:20.187960 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:20.187923 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0ae258-56f894cd47-vmwzz" event={"ID":"6fa64c39-aa67-4869-9164-a3edee1c0be8","Type":"ContainerStarted","Data":"5ad1a35d6242c4f3006835ed52c303d74ef58455486434c6f51770e9af44f6eb"} Oct 30 01:42:20.188144 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:20.187987 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0ae258-56f894cd47-vmwzz" event={"ID":"6fa64c39-aa67-4869-9164-a3edee1c0be8","Type":"ContainerStarted","Data":"6898db5b2223b918d171d9e19ab4e392282f25567034432497eec8168a0aa0be"} Oct 30 01:42:20.188144 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:20.188082 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0ae258-56f894cd47-vmwzz" Oct 30 01:42:20.226939 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:20.226882 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0ae258-56f894cd47-vmwzz" podStartSLOduration=1.226864422 podStartE2EDuration="1.226864422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:42:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:42:20.226544747 +0000 UTC m=+1463.804832076" watchObservedRunningTime="2026-10-30 01:42:20.226864422 +0000 UTC m=+1463.805151758" Oct 30 01:42:21.495524 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:42:21.495486 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:42:31.206521 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:31.206491 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0ae258-56f894cd47-vmwzz" Oct 30 01:42:32.484026 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:32.484000 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-wp6mf_fc46a03a-6581-4d32-8ed7-a4d5c53625b4/payload-processing/0.log" Oct 30 01:42:32.721167 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:32.721137 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-wp6mf_fc46a03a-6581-4d32-8ed7-a4d5c53625b4/payload-processing/0.log" Oct 30 01:42:32.889269 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:32.889235 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c9d174-5479cc575f-lc7f8"] Oct 30 01:42:32.893670 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:32.893647 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c9d174-5479cc575f-lc7f8" Oct 30 01:42:32.917569 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:32.917451 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c9d174-5479cc575f-lc7f8"] Oct 30 01:42:32.918992 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:32.918969 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-c9d174\"" Oct 30 01:42:32.919709 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:32.919687 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-c9d174-dockercfg-prgfk\"" Oct 30 01:42:32.948644 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:32.948614 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-f01cbc-77449896b-mzbkb_74c960b2-ff58-455b-8eb4-43cc4171e32c/payload-processing/0.log" Oct 30 01:42:33.060165 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:33.060128 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8cb6\" (UniqueName: \"kubernetes.io/projected/cec66c15-dfbf-4cc8-b4e7-bc42bdc45bae-kube-api-access-s8cb6\") pod \"payload-pre-processing-e2e-praxis-b-c9d174-5479cc575f-lc7f8\" (UID: \"cec66c15-dfbf-4cc8-b4e7-bc42bdc45bae\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c9d174-5479cc575f-lc7f8" Oct 30 01:42:33.060641 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:33.060617 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cec66c15-dfbf-4cc8-b4e7-bc42bdc45bae-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-c9d174-5479cc575f-lc7f8\" (UID: \"cec66c15-dfbf-4cc8-b4e7-bc42bdc45bae\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c9d174-5479cc575f-lc7f8" Oct 30 01:42:33.066603 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:33.066572 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c9d174-5f5c577c94-nf2jj"] Oct 30 01:42:33.144608 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:33.106829 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c9d174-5f5c577c94-nf2jj" Oct 30 01:42:33.167559 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:33.162667 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cec66c15-dfbf-4cc8-b4e7-bc42bdc45bae-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-c9d174-5479cc575f-lc7f8\" (UID: \"cec66c15-dfbf-4cc8-b4e7-bc42bdc45bae\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c9d174-5479cc575f-lc7f8" Oct 30 01:42:33.167559 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:33.162868 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8cb6\" (UniqueName: \"kubernetes.io/projected/cec66c15-dfbf-4cc8-b4e7-bc42bdc45bae-kube-api-access-s8cb6\") pod \"payload-pre-processing-e2e-praxis-b-c9d174-5479cc575f-lc7f8\" (UID: \"cec66c15-dfbf-4cc8-b4e7-bc42bdc45bae\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c9d174-5479cc575f-lc7f8" Oct 30 01:42:33.179922 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:33.176852 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-f01cbc-77449896b-mzbkb_74c960b2-ff58-455b-8eb4-43cc4171e32c/payload-processing/0.log" Oct 30 01:42:33.179922 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:33.178512 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cec66c15-dfbf-4cc8-b4e7-bc42bdc45bae-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-c9d174-5479cc575f-lc7f8\" (UID: \"cec66c15-dfbf-4cc8-b4e7-bc42bdc45bae\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c9d174-5479cc575f-lc7f8" Oct 30 01:42:33.188981 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:33.187527 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c9d174-5f5c577c94-nf2jj"] Oct 30 01:42:33.199610 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:33.193222 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8cb6\" (UniqueName: \"kubernetes.io/projected/cec66c15-dfbf-4cc8-b4e7-bc42bdc45bae-kube-api-access-s8cb6\") pod \"payload-pre-processing-e2e-praxis-b-c9d174-5479cc575f-lc7f8\" (UID: \"cec66c15-dfbf-4cc8-b4e7-bc42bdc45bae\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c9d174-5479cc575f-lc7f8" Oct 30 01:42:33.211837 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:33.211401 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c9d174-5479cc575f-lc7f8" Oct 30 01:42:33.263937 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:33.263904 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/597150a2-dcbe-4a84-9a68-41eb187cbce4-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-c9d174-5f5c577c94-nf2jj\" (UID: \"597150a2-dcbe-4a84-9a68-41eb187cbce4\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c9d174-5f5c577c94-nf2jj" Oct 30 01:42:33.264126 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:33.263966 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ht4lz\" (UniqueName: \"kubernetes.io/projected/597150a2-dcbe-4a84-9a68-41eb187cbce4-kube-api-access-ht4lz\") pod \"payload-processing-e2e-praxis-b-c9d174-5f5c577c94-nf2jj\" (UID: \"597150a2-dcbe-4a84-9a68-41eb187cbce4\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c9d174-5f5c577c94-nf2jj" Oct 30 01:42:33.365136 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:33.365101 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/597150a2-dcbe-4a84-9a68-41eb187cbce4-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-c9d174-5f5c577c94-nf2jj\" (UID: \"597150a2-dcbe-4a84-9a68-41eb187cbce4\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c9d174-5f5c577c94-nf2jj" Oct 30 01:42:33.365350 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:33.365152 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ht4lz\" (UniqueName: \"kubernetes.io/projected/597150a2-dcbe-4a84-9a68-41eb187cbce4-kube-api-access-ht4lz\") pod \"payload-processing-e2e-praxis-b-c9d174-5f5c577c94-nf2jj\" (UID: \"597150a2-dcbe-4a84-9a68-41eb187cbce4\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c9d174-5f5c577c94-nf2jj" Oct 30 01:42:33.365834 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:33.365814 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/597150a2-dcbe-4a84-9a68-41eb187cbce4-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-c9d174-5f5c577c94-nf2jj\" (UID: \"597150a2-dcbe-4a84-9a68-41eb187cbce4\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c9d174-5f5c577c94-nf2jj" Oct 30 01:42:33.384743 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:33.384709 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ht4lz\" (UniqueName: \"kubernetes.io/projected/597150a2-dcbe-4a84-9a68-41eb187cbce4-kube-api-access-ht4lz\") pod \"payload-processing-e2e-praxis-b-c9d174-5f5c577c94-nf2jj\" (UID: \"597150a2-dcbe-4a84-9a68-41eb187cbce4\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c9d174-5f5c577c94-nf2jj" Oct 30 01:42:33.501804 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:33.501715 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c9d174-5f5c577c94-nf2jj" Oct 30 01:42:33.511910 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:33.511881 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c9d174-5479cc575f-lc7f8"] Oct 30 01:42:33.514154 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:42:33.514123 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcec66c15_dfbf_4cc8_b4e7_bc42bdc45bae.slice/crio-18e58bdd448ddbc941c445eb6abc1cca28c1b1fd60e0d15e0899bbf36637eeae WatchSource:0}: Error finding container 18e58bdd448ddbc941c445eb6abc1cca28c1b1fd60e0d15e0899bbf36637eeae: Status 404 returned error can't find the container with id 18e58bdd448ddbc941c445eb6abc1cca28c1b1fd60e0d15e0899bbf36637eeae Oct 30 01:42:33.588478 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:33.588444 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr"] Oct 30 01:42:33.588792 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:33.588762 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr" podUID="43cff085-9d72-449b-a7f2-e9033b919e1f" containerName="istio-proxy" containerID="cri-o://f5d49aa71eb691fec202516948d393a28786182c737ab92b295f016696b957f9" gracePeriod=30 Oct 30 01:42:33.804890 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:33.804861 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c9d174-5f5c577c94-nf2jj"] Oct 30 01:42:33.807390 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:42:33.807366 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod597150a2_dcbe_4a84_9a68_41eb187cbce4.slice/crio-e898fbad79593d800eeac0399f6810c707ae08c2159a35d81a7211c2933a988f WatchSource:0}: Error finding container e898fbad79593d800eeac0399f6810c707ae08c2159a35d81a7211c2933a988f: Status 404 returned error can't find the container with id e898fbad79593d800eeac0399f6810c707ae08c2159a35d81a7211c2933a988f Oct 30 01:42:34.373607 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:34.373565 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c9d174-5479cc575f-lc7f8" event={"ID":"cec66c15-dfbf-4cc8-b4e7-bc42bdc45bae","Type":"ContainerStarted","Data":"527abbadaf9a735be77b18bc5515319a1b9e8a41a05209fd9d6771f344a41ec1"} Oct 30 01:42:34.373818 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:34.373638 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c9d174-5479cc575f-lc7f8" Oct 30 01:42:34.373818 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:34.373657 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c9d174-5479cc575f-lc7f8" event={"ID":"cec66c15-dfbf-4cc8-b4e7-bc42bdc45bae","Type":"ContainerStarted","Data":"18e58bdd448ddbc941c445eb6abc1cca28c1b1fd60e0d15e0899bbf36637eeae"} Oct 30 01:42:34.375172 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:34.375142 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c9d174-5f5c577c94-nf2jj" event={"ID":"597150a2-dcbe-4a84-9a68-41eb187cbce4","Type":"ContainerStarted","Data":"bbae8d8d4952cbe06953bb10aa0e3775cb566b6aefcedd23be743209df9fa86a"} Oct 30 01:42:34.375273 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:34.375199 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-c9d174-5f5c577c94-nf2jj" Oct 30 01:42:34.375273 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:34.375219 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c9d174-5f5c577c94-nf2jj" event={"ID":"597150a2-dcbe-4a84-9a68-41eb187cbce4","Type":"ContainerStarted","Data":"e898fbad79593d800eeac0399f6810c707ae08c2159a35d81a7211c2933a988f"} Oct 30 01:42:34.415078 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:34.415022 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c9d174-5479cc575f-lc7f8" podStartSLOduration=2.415007274 podStartE2EDuration="2.415007274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:42:34.41439305 +0000 UTC m=+1477.992680391" watchObservedRunningTime="2026-10-30 01:42:34.415007274 +0000 UTC m=+1477.993294610" Oct 30 01:42:34.454262 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:34.454209 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-c9d174-5f5c577c94-nf2jj" podStartSLOduration=2.454195891 podStartE2EDuration="2.454195891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:42:34.452586941 +0000 UTC m=+1478.030874272" watchObservedRunningTime="2026-10-30 01:42:34.454195891 +0000 UTC m=+1478.032483227" Oct 30 01:42:36.752260 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:36.752226 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:42:38.863871 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:38.863808 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr" Oct 30 01:42:38.995648 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:38.995617 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43cff085-9d72-449b-a7f2-e9033b919e1f-istio-token\" (UniqueName: \"kubernetes.io/projected/43cff085-9d72-449b-a7f2-e9033b919e1f-istio-token\") pod \"43cff085-9d72-449b-a7f2-e9033b919e1f\" (UID: \"43cff085-9d72-449b-a7f2-e9033b919e1f\") " Oct 30 01:42:38.995648 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:38.995653 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43cff085-9d72-449b-a7f2-e9033b919e1f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43cff085-9d72-449b-a7f2-e9033b919e1f-istio-envoy\") pod \"43cff085-9d72-449b-a7f2-e9033b919e1f\" (UID: \"43cff085-9d72-449b-a7f2-e9033b919e1f\") " Oct 30 01:42:38.995916 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:38.995698 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43cff085-9d72-449b-a7f2-e9033b919e1f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cff085-9d72-449b-a7f2-e9033b919e1f-workload-socket\") pod \"43cff085-9d72-449b-a7f2-e9033b919e1f\" (UID: \"43cff085-9d72-449b-a7f2-e9033b919e1f\") " Oct 30 01:42:38.995916 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:38.995734 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43cff085-9d72-449b-a7f2-e9033b919e1f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43cff085-9d72-449b-a7f2-e9033b919e1f-workload-certs\") pod \"43cff085-9d72-449b-a7f2-e9033b919e1f\" (UID: \"43cff085-9d72-449b-a7f2-e9033b919e1f\") " Oct 30 01:42:38.995916 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:38.995758 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43cff085-9d72-449b-a7f2-e9033b919e1f-kube-api-access-xk6f5\" (UniqueName: \"kubernetes.io/projected/43cff085-9d72-449b-a7f2-e9033b919e1f-kube-api-access-xk6f5\") pod \"43cff085-9d72-449b-a7f2-e9033b919e1f\" (UID: \"43cff085-9d72-449b-a7f2-e9033b919e1f\") " Oct 30 01:42:38.995916 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:38.995817 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43cff085-9d72-449b-a7f2-e9033b919e1f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43cff085-9d72-449b-a7f2-e9033b919e1f-istio-podinfo\") pod \"43cff085-9d72-449b-a7f2-e9033b919e1f\" (UID: \"43cff085-9d72-449b-a7f2-e9033b919e1f\") " Oct 30 01:42:38.996126 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:38.996029 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43cff085-9d72-449b-a7f2-e9033b919e1f-workload-socket" pod "43cff085-9d72-449b-a7f2-e9033b919e1f" (UID: "43cff085-9d72-449b-a7f2-e9033b919e1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:42:38.996126 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:38.996047 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43cff085-9d72-449b-a7f2-e9033b919e1f-workload-certs" pod "43cff085-9d72-449b-a7f2-e9033b919e1f" (UID: "43cff085-9d72-449b-a7f2-e9033b919e1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:42:38.996126 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:38.996038 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43cff085-9d72-449b-a7f2-e9033b919e1f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43cff085-9d72-449b-a7f2-e9033b919e1f-istiod-ca-cert\") pod \"43cff085-9d72-449b-a7f2-e9033b919e1f\" (UID: \"43cff085-9d72-449b-a7f2-e9033b919e1f\") " Oct 30 01:42:38.996126 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:38.996117 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43cff085-9d72-449b-a7f2-e9033b919e1f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cff085-9d72-449b-a7f2-e9033b919e1f-credential-socket\") pod \"43cff085-9d72-449b-a7f2-e9033b919e1f\" (UID: \"43cff085-9d72-449b-a7f2-e9033b919e1f\") " Oct 30 01:42:38.996328 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:38.996155 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43cff085-9d72-449b-a7f2-e9033b919e1f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43cff085-9d72-449b-a7f2-e9033b919e1f-istio-data\") pod \"43cff085-9d72-449b-a7f2-e9033b919e1f\" (UID: \"43cff085-9d72-449b-a7f2-e9033b919e1f\") " Oct 30 01:42:38.996382 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:38.996345 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43cff085-9d72-449b-a7f2-e9033b919e1f-credential-socket" pod "43cff085-9d72-449b-a7f2-e9033b919e1f" (UID: "43cff085-9d72-449b-a7f2-e9033b919e1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:42:38.996442 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:38.996376 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43cff085-9d72-449b-a7f2-e9033b919e1f-istiod-ca-cert" pod "43cff085-9d72-449b-a7f2-e9033b919e1f" (UID: "43cff085-9d72-449b-a7f2-e9033b919e1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:42:38.996502 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:38.996459 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43cff085-9d72-449b-a7f2-e9033b919e1f-istio-data" pod "43cff085-9d72-449b-a7f2-e9033b919e1f" (UID: "43cff085-9d72-449b-a7f2-e9033b919e1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:42:38.996812 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:38.996792 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43cff085-9d72-449b-a7f2-e9033b919e1f-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:42:38.996812 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:38.996812 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cff085-9d72-449b-a7f2-e9033b919e1f-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:42:38.996972 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:38.996822 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43cff085-9d72-449b-a7f2-e9033b919e1f-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:42:38.996972 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:38.996831 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cff085-9d72-449b-a7f2-e9033b919e1f-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:42:38.996972 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:38.996838 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43cff085-9d72-449b-a7f2-e9033b919e1f-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:42:38.998116 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:38.998095 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43cff085-9d72-449b-a7f2-e9033b919e1f-istio-envoy" pod "43cff085-9d72-449b-a7f2-e9033b919e1f" (UID: "43cff085-9d72-449b-a7f2-e9033b919e1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:42:38.998207 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:38.998115 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43cff085-9d72-449b-a7f2-e9033b919e1f-istio-token" pod "43cff085-9d72-449b-a7f2-e9033b919e1f" (UID: "43cff085-9d72-449b-a7f2-e9033b919e1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:42:38.998353 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:38.998328 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43cff085-9d72-449b-a7f2-e9033b919e1f-kube-api-access-xk6f5" pod "43cff085-9d72-449b-a7f2-e9033b919e1f" (UID: "43cff085-9d72-449b-a7f2-e9033b919e1f"). InnerVolumeSpecName "kube-api-access-xk6f5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:42:38.998451 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:38.998351 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43cff085-9d72-449b-a7f2-e9033b919e1f-istio-podinfo" pod "43cff085-9d72-449b-a7f2-e9033b919e1f" (UID: "43cff085-9d72-449b-a7f2-e9033b919e1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:42:39.098365 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:39.098330 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk6f5\" (UniqueName: \"kubernetes.io/projected/43cff085-9d72-449b-a7f2-e9033b919e1f-kube-api-access-xk6f5\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:42:39.098365 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:39.098363 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43cff085-9d72-449b-a7f2-e9033b919e1f-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:42:39.098365 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:39.098372 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43cff085-9d72-449b-a7f2-e9033b919e1f-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:42:39.098643 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:39.098381 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43cff085-9d72-449b-a7f2-e9033b919e1f-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:42:39.460934 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:39.460893 6221 generic.go:356] "Generic (PLEG): container finished" podID="43cff085-9d72-449b-a7f2-e9033b919e1f" containerID="f5d49aa71eb691fec202516948d393a28786182c737ab92b295f016696b957f9" exitCode=0 Oct 30 01:42:39.461111 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:39.461037 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr" event={"ID":"43cff085-9d72-449b-a7f2-e9033b919e1f","Type":"ContainerDied","Data":"f5d49aa71eb691fec202516948d393a28786182c737ab92b295f016696b957f9"} Oct 30 01:42:39.461156 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:39.461120 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr" event={"ID":"43cff085-9d72-449b-a7f2-e9033b919e1f","Type":"ContainerDied","Data":"c5b5b8328038d0f6b051b15976ba86fdb31c87adbb35b784055f49ee394f38ef"} Oct 30 01:42:39.461156 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:39.461130 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr" Oct 30 01:42:39.461156 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:39.461149 6221 scope.go:122] "RemoveContainer" containerID="f5d49aa71eb691fec202516948d393a28786182c737ab92b295f016696b957f9" Oct 30 01:42:39.471592 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:39.471568 6221 scope.go:122] "RemoveContainer" containerID="f5d49aa71eb691fec202516948d393a28786182c737ab92b295f016696b957f9" Oct 30 01:42:39.471838 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:42:39.471818 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5d49aa71eb691fec202516948d393a28786182c737ab92b295f016696b957f9\": container with ID starting with f5d49aa71eb691fec202516948d393a28786182c737ab92b295f016696b957f9 not found: ID does not exist" containerID="f5d49aa71eb691fec202516948d393a28786182c737ab92b295f016696b957f9" Oct 30 01:42:39.471884 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:39.471847 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5d49aa71eb691fec202516948d393a28786182c737ab92b295f016696b957f9"} err="failed to get container status \"f5d49aa71eb691fec202516948d393a28786182c737ab92b295f016696b957f9\": rpc error: code = NotFound desc = could not find container \"f5d49aa71eb691fec202516948d393a28786182c737ab92b295f016696b957f9\": container with ID starting with f5d49aa71eb691fec202516948d393a28786182c737ab92b295f016696b957f9 not found: ID does not exist" Oct 30 01:42:39.529379 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:39.529349 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr"] Oct 30 01:42:39.541646 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:39.541621 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-ctpgr"] Oct 30 01:42:40.548334 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:40.548300 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43cff085-9d72-449b-a7f2-e9033b919e1f" path="/var/lib/kubelet/pods/43cff085-9d72-449b-a7f2-e9033b919e1f/volumes" Oct 30 01:42:45.384661 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:45.384628 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c9d174-5479cc575f-lc7f8" Oct 30 01:42:45.385104 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:45.385003 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-c9d174-5f5c577c94-nf2jj" Oct 30 01:42:51.496008 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:42:51.495970 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:42:56.722839 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:56.722810 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2vsns_d332bbb4-c699-44e8-96ba-516d8ca03e95/ovn-acl-logging/0.log" Oct 30 01:42:56.736911 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:56.736883 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2vsns_d332bbb4-c699-44e8-96ba-516d8ca03e95/ovn-acl-logging/0.log" Oct 30 01:42:59.316891 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:59.316861 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-be8a7d-67797b4964-srsbd"] Oct 30 01:42:59.317464 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:59.317447 6221 cpu_manager.go:414] "containerMap: removing container" podUID="43cff085-9d72-449b-a7f2-e9033b919e1f" containerName="istio-proxy" Oct 30 01:42:59.317511 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:59.317464 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43cff085-9d72-449b-a7f2-e9033b919e1f" containerName="istio-proxy" Oct 30 01:42:59.317638 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:59.317628 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="43cff085-9d72-449b-a7f2-e9033b919e1f" containerName="istio-proxy" Oct 30 01:42:59.322696 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:59.322674 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-be8a7d-67797b4964-srsbd" Oct 30 01:42:59.337611 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:59.337582 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-be8a7d\"" Oct 30 01:42:59.337769 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:59.337658 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-be8a7d-dockercfg-fdjc9\"" Oct 30 01:42:59.340721 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:59.340693 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-be8a7d-67797b4964-srsbd"] Oct 30 01:42:59.372189 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:59.372153 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-be8a7d-5b68d984ff-5bh77"] Oct 30 01:42:59.377114 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:59.377088 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-be8a7d-5b68d984ff-5bh77" Oct 30 01:42:59.391442 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:59.391407 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-be8a7d-5b68d984ff-5bh77"] Oct 30 01:42:59.537556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:59.531931 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ec92517-e71d-4a1c-a142-000fd54b090c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-be8a7d-5b68d984ff-5bh77\" (UID: \"7ec92517-e71d-4a1c-a142-000fd54b090c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-be8a7d-5b68d984ff-5bh77" Oct 30 01:42:59.537887 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:59.537855 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx56s\" (UniqueName: \"kubernetes.io/projected/7ec92517-e71d-4a1c-a142-000fd54b090c-kube-api-access-mx56s\") pod \"payload-pre-processing-e2e-praxis-b-be8a7d-5b68d984ff-5bh77\" (UID: \"7ec92517-e71d-4a1c-a142-000fd54b090c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-be8a7d-5b68d984ff-5bh77" Oct 30 01:42:59.548564 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:59.538286 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3429734-086d-499f-b92c-28c980fdbd66-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-be8a7d-67797b4964-srsbd\" (UID: \"b3429734-086d-499f-b92c-28c980fdbd66\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-be8a7d-67797b4964-srsbd" Oct 30 01:42:59.588233 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:59.588130 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phnw9\" (UniqueName: \"kubernetes.io/projected/b3429734-086d-499f-b92c-28c980fdbd66-kube-api-access-phnw9\") pod \"payload-processing-e2e-praxis-b-be8a7d-67797b4964-srsbd\" (UID: \"b3429734-086d-499f-b92c-28c980fdbd66\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-be8a7d-67797b4964-srsbd" Oct 30 01:42:59.690262 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:59.690218 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3429734-086d-499f-b92c-28c980fdbd66-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-be8a7d-67797b4964-srsbd\" (UID: \"b3429734-086d-499f-b92c-28c980fdbd66\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-be8a7d-67797b4964-srsbd" Oct 30 01:42:59.690452 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:59.690307 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phnw9\" (UniqueName: \"kubernetes.io/projected/b3429734-086d-499f-b92c-28c980fdbd66-kube-api-access-phnw9\") pod \"payload-processing-e2e-praxis-b-be8a7d-67797b4964-srsbd\" (UID: \"b3429734-086d-499f-b92c-28c980fdbd66\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-be8a7d-67797b4964-srsbd" Oct 30 01:42:59.690452 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:59.690383 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ec92517-e71d-4a1c-a142-000fd54b090c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-be8a7d-5b68d984ff-5bh77\" (UID: \"7ec92517-e71d-4a1c-a142-000fd54b090c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-be8a7d-5b68d984ff-5bh77" Oct 30 01:42:59.690452 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:59.690411 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx56s\" (UniqueName: \"kubernetes.io/projected/7ec92517-e71d-4a1c-a142-000fd54b090c-kube-api-access-mx56s\") pod \"payload-pre-processing-e2e-praxis-b-be8a7d-5b68d984ff-5bh77\" (UID: \"7ec92517-e71d-4a1c-a142-000fd54b090c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-be8a7d-5b68d984ff-5bh77" Oct 30 01:42:59.691046 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:59.691021 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3429734-086d-499f-b92c-28c980fdbd66-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-be8a7d-67797b4964-srsbd\" (UID: \"b3429734-086d-499f-b92c-28c980fdbd66\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-be8a7d-67797b4964-srsbd" Oct 30 01:42:59.691195 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:59.691175 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ec92517-e71d-4a1c-a142-000fd54b090c-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-be8a7d-5b68d984ff-5bh77\" (UID: \"7ec92517-e71d-4a1c-a142-000fd54b090c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-be8a7d-5b68d984ff-5bh77" Oct 30 01:42:59.710464 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:59.710430 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx56s\" (UniqueName: \"kubernetes.io/projected/7ec92517-e71d-4a1c-a142-000fd54b090c-kube-api-access-mx56s\") pod \"payload-pre-processing-e2e-praxis-b-be8a7d-5b68d984ff-5bh77\" (UID: \"7ec92517-e71d-4a1c-a142-000fd54b090c\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-be8a7d-5b68d984ff-5bh77" Oct 30 01:42:59.710667 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:59.710430 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phnw9\" (UniqueName: \"kubernetes.io/projected/b3429734-086d-499f-b92c-28c980fdbd66-kube-api-access-phnw9\") pod \"payload-processing-e2e-praxis-b-be8a7d-67797b4964-srsbd\" (UID: \"b3429734-086d-499f-b92c-28c980fdbd66\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-be8a7d-67797b4964-srsbd" Oct 30 01:42:59.936339 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:59.936239 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-be8a7d-67797b4964-srsbd" Oct 30 01:42:59.997557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:42:59.997118 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-be8a7d-5b68d984ff-5bh77" Oct 30 01:43:00.238065 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:00.238022 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-be8a7d-67797b4964-srsbd"] Oct 30 01:43:00.240503 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:43:00.240478 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3429734_086d_499f_b92c_28c980fdbd66.slice/crio-333f1955b084b9c7337c041a5592a4167a2773b6897ca4f2fbd7e65a3d0905a4 WatchSource:0}: Error finding container 333f1955b084b9c7337c041a5592a4167a2773b6897ca4f2fbd7e65a3d0905a4: Status 404 returned error can't find the container with id 333f1955b084b9c7337c041a5592a4167a2773b6897ca4f2fbd7e65a3d0905a4 Oct 30 01:43:00.333032 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:00.332754 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-be8a7d-5b68d984ff-5bh77"] Oct 30 01:43:00.355964 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:43:00.355932 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ec92517_e71d_4a1c_a142_000fd54b090c.slice/crio-a3e41a47594c3bcdf3b53bfe2db32819b6ed2ab6575c68ade82d197999b40cae WatchSource:0}: Error finding container a3e41a47594c3bcdf3b53bfe2db32819b6ed2ab6575c68ade82d197999b40cae: Status 404 returned error can't find the container with id a3e41a47594c3bcdf3b53bfe2db32819b6ed2ab6575c68ade82d197999b40cae Oct 30 01:43:00.649393 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:00.649351 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-be8a7d-5b68d984ff-5bh77" event={"ID":"7ec92517-e71d-4a1c-a142-000fd54b090c","Type":"ContainerStarted","Data":"7756bf99bbe93f8131811aeb7e29a32f76dd90b91ca02629123302dfcb3512ef"} Oct 30 01:43:00.649700 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:00.649421 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-be8a7d-5b68d984ff-5bh77" event={"ID":"7ec92517-e71d-4a1c-a142-000fd54b090c","Type":"ContainerStarted","Data":"a3e41a47594c3bcdf3b53bfe2db32819b6ed2ab6575c68ade82d197999b40cae"} Oct 30 01:43:00.649700 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:00.649572 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-be8a7d-5b68d984ff-5bh77" Oct 30 01:43:00.651174 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:00.651147 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-be8a7d-67797b4964-srsbd" event={"ID":"b3429734-086d-499f-b92c-28c980fdbd66","Type":"ContainerStarted","Data":"3fa9f453f1473361626ee9004d2318f0440f9c779b4a534fd98364d93f27a32b"} Oct 30 01:43:00.651294 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:00.651198 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-be8a7d-67797b4964-srsbd" event={"ID":"b3429734-086d-499f-b92c-28c980fdbd66","Type":"ContainerStarted","Data":"333f1955b084b9c7337c041a5592a4167a2773b6897ca4f2fbd7e65a3d0905a4"} Oct 30 01:43:00.651294 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:00.651229 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-be8a7d-67797b4964-srsbd" Oct 30 01:43:00.691023 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:00.690974 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-be8a7d-5b68d984ff-5bh77" podStartSLOduration=1.690959844 podStartE2EDuration="1.690959844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:42:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:43:00.688217219 +0000 UTC m=+1504.266504556" watchObservedRunningTime="2026-10-30 01:43:00.690959844 +0000 UTC m=+1504.269247180" Oct 30 01:43:00.729161 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:00.729105 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-be8a7d-67797b4964-srsbd" podStartSLOduration=1.729090376 podStartE2EDuration="1.729090376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:42:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:43:00.725743216 +0000 UTC m=+1504.304030553" watchObservedRunningTime="2026-10-30 01:43:00.729090376 +0000 UTC m=+1504.307377711" Oct 30 01:43:11.660360 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:11.660332 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-be8a7d-67797b4964-srsbd" Oct 30 01:43:11.660770 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:11.660397 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-be8a7d-5b68d984ff-5bh77" Oct 30 01:43:12.649705 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:12.649668 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz"] Oct 30 01:43:12.649956 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:12.649917 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" podUID="32567804-08a2-48d7-942d-bd82afff5728" containerName="istio-proxy" containerID="cri-o://2c4b408325e40d844e2372d21309d0995f2b4aa0ed497ac2eb02c3f4fe6ba759" gracePeriod=30 Oct 30 01:43:13.783163 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:13.783117 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx"] Oct 30 01:43:13.784070 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:13.784003 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx" podUID="324f43dc-720f-4b56-9f26-39ffce236212" containerName="istio-proxy" containerID="cri-o://2b26eec3041b65c7b139e454733faeed578314ece02914c03b4cffad09e947b2" gracePeriod=30 Oct 30 01:43:16.109554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:16.108927 6221 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:43:16.109554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:16.109000 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" podUID="32567804-08a2-48d7-942d-bd82afff5728" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:43:17.970906 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:17.970875 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" Oct 30 01:43:18.056442 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:18.056399 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32567804-08a2-48d7-942d-bd82afff5728-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32567804-08a2-48d7-942d-bd82afff5728-workload-certs\") pod \"32567804-08a2-48d7-942d-bd82afff5728\" (UID: \"32567804-08a2-48d7-942d-bd82afff5728\") " Oct 30 01:43:18.056662 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:18.056484 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32567804-08a2-48d7-942d-bd82afff5728-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32567804-08a2-48d7-942d-bd82afff5728-credential-socket\") pod \"32567804-08a2-48d7-942d-bd82afff5728\" (UID: \"32567804-08a2-48d7-942d-bd82afff5728\") " Oct 30 01:43:18.056662 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:18.056618 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32567804-08a2-48d7-942d-bd82afff5728-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32567804-08a2-48d7-942d-bd82afff5728-workload-socket\") pod \"32567804-08a2-48d7-942d-bd82afff5728\" (UID: \"32567804-08a2-48d7-942d-bd82afff5728\") " Oct 30 01:43:18.056662 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:18.056661 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32567804-08a2-48d7-942d-bd82afff5728-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32567804-08a2-48d7-942d-bd82afff5728-istio-podinfo\") pod \"32567804-08a2-48d7-942d-bd82afff5728\" (UID: \"32567804-08a2-48d7-942d-bd82afff5728\") " Oct 30 01:43:18.056844 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:18.056696 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32567804-08a2-48d7-942d-bd82afff5728-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32567804-08a2-48d7-942d-bd82afff5728-istiod-ca-cert\") pod \"32567804-08a2-48d7-942d-bd82afff5728\" (UID: \"32567804-08a2-48d7-942d-bd82afff5728\") " Oct 30 01:43:18.056844 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:18.056745 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32567804-08a2-48d7-942d-bd82afff5728-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32567804-08a2-48d7-942d-bd82afff5728-istio-envoy\") pod \"32567804-08a2-48d7-942d-bd82afff5728\" (UID: \"32567804-08a2-48d7-942d-bd82afff5728\") " Oct 30 01:43:18.056844 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:18.056754 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32567804-08a2-48d7-942d-bd82afff5728-workload-certs" pod "32567804-08a2-48d7-942d-bd82afff5728" (UID: "32567804-08a2-48d7-942d-bd82afff5728"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:43:18.056844 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:18.056803 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32567804-08a2-48d7-942d-bd82afff5728-kube-api-access-vnwkj\" (UniqueName: \"kubernetes.io/projected/32567804-08a2-48d7-942d-bd82afff5728-kube-api-access-vnwkj\") pod \"32567804-08a2-48d7-942d-bd82afff5728\" (UID: \"32567804-08a2-48d7-942d-bd82afff5728\") " Oct 30 01:43:18.057077 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:18.056873 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32567804-08a2-48d7-942d-bd82afff5728-credential-socket" pod "32567804-08a2-48d7-942d-bd82afff5728" (UID: "32567804-08a2-48d7-942d-bd82afff5728"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:43:18.057077 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:18.056911 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32567804-08a2-48d7-942d-bd82afff5728-workload-socket" pod "32567804-08a2-48d7-942d-bd82afff5728" (UID: "32567804-08a2-48d7-942d-bd82afff5728"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:43:18.057077 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:18.057009 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32567804-08a2-48d7-942d-bd82afff5728-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32567804-08a2-48d7-942d-bd82afff5728-istio-data\") pod \"32567804-08a2-48d7-942d-bd82afff5728\" (UID: \"32567804-08a2-48d7-942d-bd82afff5728\") " Oct 30 01:43:18.057077 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:18.057052 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32567804-08a2-48d7-942d-bd82afff5728-istio-token\" (UniqueName: \"kubernetes.io/projected/32567804-08a2-48d7-942d-bd82afff5728-istio-token\") pod \"32567804-08a2-48d7-942d-bd82afff5728\" (UID: \"32567804-08a2-48d7-942d-bd82afff5728\") " Oct 30 01:43:18.057298 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:18.057092 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32567804-08a2-48d7-942d-bd82afff5728-istiod-ca-cert" pod "32567804-08a2-48d7-942d-bd82afff5728" (UID: "32567804-08a2-48d7-942d-bd82afff5728"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:43:18.057472 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:18.057447 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32567804-08a2-48d7-942d-bd82afff5728-istio-data" pod "32567804-08a2-48d7-942d-bd82afff5728" (UID: "32567804-08a2-48d7-942d-bd82afff5728"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:43:18.057725 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:18.057695 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32567804-08a2-48d7-942d-bd82afff5728-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:18.057725 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:18.057717 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32567804-08a2-48d7-942d-bd82afff5728-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:18.057853 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:18.057730 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32567804-08a2-48d7-942d-bd82afff5728-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:18.057853 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:18.057743 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32567804-08a2-48d7-942d-bd82afff5728-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:18.059656 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:18.059612 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32567804-08a2-48d7-942d-bd82afff5728-istio-podinfo" pod "32567804-08a2-48d7-942d-bd82afff5728" (UID: "32567804-08a2-48d7-942d-bd82afff5728"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:43:18.060025 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:18.060002 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32567804-08a2-48d7-942d-bd82afff5728-istio-token" pod "32567804-08a2-48d7-942d-bd82afff5728" (UID: "32567804-08a2-48d7-942d-bd82afff5728"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:43:18.060576 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:18.060523 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32567804-08a2-48d7-942d-bd82afff5728-istio-envoy" pod "32567804-08a2-48d7-942d-bd82afff5728" (UID: "32567804-08a2-48d7-942d-bd82afff5728"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:43:18.060576 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:18.060558 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32567804-08a2-48d7-942d-bd82afff5728-kube-api-access-vnwkj" pod "32567804-08a2-48d7-942d-bd82afff5728" (UID: "32567804-08a2-48d7-942d-bd82afff5728"). InnerVolumeSpecName "kube-api-access-vnwkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:43:18.159328 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:18.159293 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32567804-08a2-48d7-942d-bd82afff5728-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:18.159328 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:18.159325 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32567804-08a2-48d7-942d-bd82afff5728-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:18.159328 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:18.159335 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnwkj\" (UniqueName: \"kubernetes.io/projected/32567804-08a2-48d7-942d-bd82afff5728-kube-api-access-vnwkj\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:18.159328 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:18.159345 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32567804-08a2-48d7-942d-bd82afff5728-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:18.159670 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:18.159353 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32567804-08a2-48d7-942d-bd82afff5728-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:18.770806 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:18.770771 6221 generic.go:356] "Generic (PLEG): container finished" podID="32567804-08a2-48d7-942d-bd82afff5728" containerID="2c4b408325e40d844e2372d21309d0995f2b4aa0ed497ac2eb02c3f4fe6ba759" exitCode=0 Oct 30 01:43:18.770999 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:18.770818 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" Oct 30 01:43:18.770999 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:18.770851 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" event={"ID":"32567804-08a2-48d7-942d-bd82afff5728","Type":"ContainerDied","Data":"2c4b408325e40d844e2372d21309d0995f2b4aa0ed497ac2eb02c3f4fe6ba759"} Oct 30 01:43:18.770999 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:18.770906 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz" event={"ID":"32567804-08a2-48d7-942d-bd82afff5728","Type":"ContainerDied","Data":"f0cc7ae573acc454605d213778a39320000b312542838189c616d37aacc46df0"} Oct 30 01:43:18.770999 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:18.770931 6221 scope.go:122] "RemoveContainer" containerID="2c4b408325e40d844e2372d21309d0995f2b4aa0ed497ac2eb02c3f4fe6ba759" Oct 30 01:43:18.780608 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:18.780591 6221 scope.go:122] "RemoveContainer" containerID="6c8adf670d2a303004f1d2686d816378502841ffc9b15f96b962fd2356da0d73" Oct 30 01:43:18.789347 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:18.789331 6221 scope.go:122] "RemoveContainer" containerID="2c4b408325e40d844e2372d21309d0995f2b4aa0ed497ac2eb02c3f4fe6ba759" Oct 30 01:43:18.789747 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:43:18.789722 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c4b408325e40d844e2372d21309d0995f2b4aa0ed497ac2eb02c3f4fe6ba759\": container with ID starting with 2c4b408325e40d844e2372d21309d0995f2b4aa0ed497ac2eb02c3f4fe6ba759 not found: ID does not exist" containerID="2c4b408325e40d844e2372d21309d0995f2b4aa0ed497ac2eb02c3f4fe6ba759" Oct 30 01:43:18.789808 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:18.789756 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c4b408325e40d844e2372d21309d0995f2b4aa0ed497ac2eb02c3f4fe6ba759"} err="failed to get container status \"2c4b408325e40d844e2372d21309d0995f2b4aa0ed497ac2eb02c3f4fe6ba759\": rpc error: code = NotFound desc = could not find container \"2c4b408325e40d844e2372d21309d0995f2b4aa0ed497ac2eb02c3f4fe6ba759\": container with ID starting with 2c4b408325e40d844e2372d21309d0995f2b4aa0ed497ac2eb02c3f4fe6ba759 not found: ID does not exist" Oct 30 01:43:18.789808 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:18.789772 6221 scope.go:122] "RemoveContainer" containerID="6c8adf670d2a303004f1d2686d816378502841ffc9b15f96b962fd2356da0d73" Oct 30 01:43:18.790028 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:43:18.790010 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c8adf670d2a303004f1d2686d816378502841ffc9b15f96b962fd2356da0d73\": container with ID starting with 6c8adf670d2a303004f1d2686d816378502841ffc9b15f96b962fd2356da0d73 not found: ID does not exist" containerID="6c8adf670d2a303004f1d2686d816378502841ffc9b15f96b962fd2356da0d73" Oct 30 01:43:18.790078 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:18.790033 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c8adf670d2a303004f1d2686d816378502841ffc9b15f96b962fd2356da0d73"} err="failed to get container status \"6c8adf670d2a303004f1d2686d816378502841ffc9b15f96b962fd2356da0d73\": rpc error: code = NotFound desc = could not find container \"6c8adf670d2a303004f1d2686d816378502841ffc9b15f96b962fd2356da0d73\": container with ID starting with 6c8adf670d2a303004f1d2686d816378502841ffc9b15f96b962fd2356da0d73 not found: ID does not exist" Oct 30 01:43:18.836102 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:18.836074 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz"] Oct 30 01:43:18.848126 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:18.848104 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-nrmdz"] Oct 30 01:43:19.053769 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:19.053741 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx" Oct 30 01:43:19.070375 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:19.070296 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/324f43dc-720f-4b56-9f26-39ffce236212-istio-token\" (UniqueName: \"kubernetes.io/projected/324f43dc-720f-4b56-9f26-39ffce236212-istio-token\") pod \"324f43dc-720f-4b56-9f26-39ffce236212\" (UID: \"324f43dc-720f-4b56-9f26-39ffce236212\") " Oct 30 01:43:19.070547 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:19.070389 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/324f43dc-720f-4b56-9f26-39ffce236212-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/324f43dc-720f-4b56-9f26-39ffce236212-istio-podinfo\") pod \"324f43dc-720f-4b56-9f26-39ffce236212\" (UID: \"324f43dc-720f-4b56-9f26-39ffce236212\") " Oct 30 01:43:19.070547 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:19.070422 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/324f43dc-720f-4b56-9f26-39ffce236212-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/324f43dc-720f-4b56-9f26-39ffce236212-credential-socket\") pod \"324f43dc-720f-4b56-9f26-39ffce236212\" (UID: \"324f43dc-720f-4b56-9f26-39ffce236212\") " Oct 30 01:43:19.070547 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:19.070458 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/324f43dc-720f-4b56-9f26-39ffce236212-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/324f43dc-720f-4b56-9f26-39ffce236212-istio-envoy\") pod \"324f43dc-720f-4b56-9f26-39ffce236212\" (UID: \"324f43dc-720f-4b56-9f26-39ffce236212\") " Oct 30 01:43:19.070547 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:19.070512 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/324f43dc-720f-4b56-9f26-39ffce236212-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/324f43dc-720f-4b56-9f26-39ffce236212-istio-data\") pod \"324f43dc-720f-4b56-9f26-39ffce236212\" (UID: \"324f43dc-720f-4b56-9f26-39ffce236212\") " Oct 30 01:43:19.070774 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:19.070687 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/324f43dc-720f-4b56-9f26-39ffce236212-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/324f43dc-720f-4b56-9f26-39ffce236212-workload-certs\") pod \"324f43dc-720f-4b56-9f26-39ffce236212\" (UID: \"324f43dc-720f-4b56-9f26-39ffce236212\") " Oct 30 01:43:19.070774 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:19.070744 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/324f43dc-720f-4b56-9f26-39ffce236212-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/324f43dc-720f-4b56-9f26-39ffce236212-istiod-ca-cert\") pod \"324f43dc-720f-4b56-9f26-39ffce236212\" (UID: \"324f43dc-720f-4b56-9f26-39ffce236212\") " Oct 30 01:43:19.070774 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:19.070753 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/324f43dc-720f-4b56-9f26-39ffce236212-credential-socket" pod "324f43dc-720f-4b56-9f26-39ffce236212" (UID: "324f43dc-720f-4b56-9f26-39ffce236212"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:43:19.070943 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:19.070783 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/324f43dc-720f-4b56-9f26-39ffce236212-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/324f43dc-720f-4b56-9f26-39ffce236212-workload-socket\") pod \"324f43dc-720f-4b56-9f26-39ffce236212\" (UID: \"324f43dc-720f-4b56-9f26-39ffce236212\") " Oct 30 01:43:19.070943 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:19.070895 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/324f43dc-720f-4b56-9f26-39ffce236212-kube-api-access-z88h9\" (UniqueName: \"kubernetes.io/projected/324f43dc-720f-4b56-9f26-39ffce236212-kube-api-access-z88h9\") pod \"324f43dc-720f-4b56-9f26-39ffce236212\" (UID: \"324f43dc-720f-4b56-9f26-39ffce236212\") " Oct 30 01:43:19.071052 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:19.070993 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/324f43dc-720f-4b56-9f26-39ffce236212-workload-socket" pod "324f43dc-720f-4b56-9f26-39ffce236212" (UID: "324f43dc-720f-4b56-9f26-39ffce236212"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:43:19.071052 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:19.071013 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/324f43dc-720f-4b56-9f26-39ffce236212-workload-certs" pod "324f43dc-720f-4b56-9f26-39ffce236212" (UID: "324f43dc-720f-4b56-9f26-39ffce236212"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:43:19.071162 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:19.071100 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/324f43dc-720f-4b56-9f26-39ffce236212-istiod-ca-cert" pod "324f43dc-720f-4b56-9f26-39ffce236212" (UID: "324f43dc-720f-4b56-9f26-39ffce236212"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:43:19.071214 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:19.071171 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/324f43dc-720f-4b56-9f26-39ffce236212-istio-data" pod "324f43dc-720f-4b56-9f26-39ffce236212" (UID: "324f43dc-720f-4b56-9f26-39ffce236212"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:43:19.071680 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:19.071656 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/324f43dc-720f-4b56-9f26-39ffce236212-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:19.071796 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:19.071684 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/324f43dc-720f-4b56-9f26-39ffce236212-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:19.071796 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:19.071699 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/324f43dc-720f-4b56-9f26-39ffce236212-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:19.071796 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:19.071715 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/324f43dc-720f-4b56-9f26-39ffce236212-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:19.071796 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:19.071729 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/324f43dc-720f-4b56-9f26-39ffce236212-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:19.072811 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:19.072784 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/324f43dc-720f-4b56-9f26-39ffce236212-istio-token" pod "324f43dc-720f-4b56-9f26-39ffce236212" (UID: "324f43dc-720f-4b56-9f26-39ffce236212"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:43:19.072910 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:19.072808 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/324f43dc-720f-4b56-9f26-39ffce236212-istio-podinfo" pod "324f43dc-720f-4b56-9f26-39ffce236212" (UID: "324f43dc-720f-4b56-9f26-39ffce236212"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:43:19.072910 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:19.072815 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/324f43dc-720f-4b56-9f26-39ffce236212-istio-envoy" pod "324f43dc-720f-4b56-9f26-39ffce236212" (UID: "324f43dc-720f-4b56-9f26-39ffce236212"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:43:19.072997 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:19.072940 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/324f43dc-720f-4b56-9f26-39ffce236212-kube-api-access-z88h9" pod "324f43dc-720f-4b56-9f26-39ffce236212" (UID: "324f43dc-720f-4b56-9f26-39ffce236212"). InnerVolumeSpecName "kube-api-access-z88h9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:43:19.173523 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:19.173490 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/324f43dc-720f-4b56-9f26-39ffce236212-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:19.173523 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:19.173520 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/324f43dc-720f-4b56-9f26-39ffce236212-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:19.173523 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:19.173541 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z88h9\" (UniqueName: \"kubernetes.io/projected/324f43dc-720f-4b56-9f26-39ffce236212-kube-api-access-z88h9\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:19.173801 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:19.173556 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/324f43dc-720f-4b56-9f26-39ffce236212-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:19.778991 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:19.778960 6221 generic.go:356] "Generic (PLEG): container finished" podID="324f43dc-720f-4b56-9f26-39ffce236212" containerID="2b26eec3041b65c7b139e454733faeed578314ece02914c03b4cffad09e947b2" exitCode=0 Oct 30 01:43:19.779173 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:19.779005 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx" Oct 30 01:43:19.779173 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:19.779042 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx" event={"ID":"324f43dc-720f-4b56-9f26-39ffce236212","Type":"ContainerDied","Data":"2b26eec3041b65c7b139e454733faeed578314ece02914c03b4cffad09e947b2"} Oct 30 01:43:19.779173 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:19.779097 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx" event={"ID":"324f43dc-720f-4b56-9f26-39ffce236212","Type":"ContainerDied","Data":"d1a9147c028e0541a000372997f4fbad13762e8a9bc7c3c730db1e43ad43568b"} Oct 30 01:43:19.779173 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:19.779126 6221 scope.go:122] "RemoveContainer" containerID="2b26eec3041b65c7b139e454733faeed578314ece02914c03b4cffad09e947b2" Oct 30 01:43:19.789373 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:19.789357 6221 scope.go:122] "RemoveContainer" containerID="2b26eec3041b65c7b139e454733faeed578314ece02914c03b4cffad09e947b2" Oct 30 01:43:19.789684 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:43:19.789666 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b26eec3041b65c7b139e454733faeed578314ece02914c03b4cffad09e947b2\": container with ID starting with 2b26eec3041b65c7b139e454733faeed578314ece02914c03b4cffad09e947b2 not found: ID does not exist" containerID="2b26eec3041b65c7b139e454733faeed578314ece02914c03b4cffad09e947b2" Oct 30 01:43:19.789766 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:19.789696 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b26eec3041b65c7b139e454733faeed578314ece02914c03b4cffad09e947b2"} err="failed to get container status \"2b26eec3041b65c7b139e454733faeed578314ece02914c03b4cffad09e947b2\": rpc error: code = NotFound desc = could not find container \"2b26eec3041b65c7b139e454733faeed578314ece02914c03b4cffad09e947b2\": container with ID starting with 2b26eec3041b65c7b139e454733faeed578314ece02914c03b4cffad09e947b2 not found: ID does not exist" Oct 30 01:43:19.832199 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:19.832172 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx"] Oct 30 01:43:19.844611 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:19.844583 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-vhvmx"] Oct 30 01:43:20.547857 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:20.547820 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="324f43dc-720f-4b56-9f26-39ffce236212" path="/var/lib/kubelet/pods/324f43dc-720f-4b56-9f26-39ffce236212/volumes" Oct 30 01:43:20.548363 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:20.548348 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32567804-08a2-48d7-942d-bd82afff5728" path="/var/lib/kubelet/pods/32567804-08a2-48d7-942d-bd82afff5728/volumes" Oct 30 01:43:21.043603 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:21.043568 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:43:21.128511 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:21.128479 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-58632b-fd88c74d7-rmhvd"] Oct 30 01:43:21.129133 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:21.129119 6221 cpu_manager.go:414] "containerMap: removing container" podUID="32567804-08a2-48d7-942d-bd82afff5728" containerName="istio-proxy" Oct 30 01:43:21.129187 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:21.129137 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32567804-08a2-48d7-942d-bd82afff5728" containerName="istio-proxy" Oct 30 01:43:21.129187 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:21.129151 6221 cpu_manager.go:414] "containerMap: removing container" podUID="324f43dc-720f-4b56-9f26-39ffce236212" containerName="istio-proxy" Oct 30 01:43:21.129187 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:21.129157 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="324f43dc-720f-4b56-9f26-39ffce236212" containerName="istio-proxy" Oct 30 01:43:21.129286 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:21.129258 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="324f43dc-720f-4b56-9f26-39ffce236212" containerName="istio-proxy" Oct 30 01:43:21.129286 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:21.129270 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="32567804-08a2-48d7-942d-bd82afff5728" containerName="istio-proxy" Oct 30 01:43:21.129286 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:21.129282 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="32567804-08a2-48d7-942d-bd82afff5728" containerName="istio-proxy" Oct 30 01:43:21.133931 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:21.133910 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-58632b-fd88c74d7-rmhvd" Oct 30 01:43:21.150497 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:21.150453 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-58632b\"" Oct 30 01:43:21.150680 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:21.150638 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-58632b-dockercfg-2pw8d\"" Oct 30 01:43:21.154726 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:21.154697 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-58632b-fd88c74d7-rmhvd"] Oct 30 01:43:21.252563 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:21.239655 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-58632b-78dbfbc54b-857kc"] Oct 30 01:43:21.338725 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:21.309737 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5b407f8-baf2-467d-817f-ccfc276df9eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-58632b-fd88c74d7-rmhvd\" (UID: \"f5b407f8-baf2-467d-817f-ccfc276df9eb\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-58632b-fd88c74d7-rmhvd" Oct 30 01:43:21.339115 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:21.339088 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw54m\" (UniqueName: \"kubernetes.io/projected/f5b407f8-baf2-467d-817f-ccfc276df9eb-kube-api-access-jw54m\") pod \"payload-pre-processing-e2e-praxis-b-58632b-fd88c74d7-rmhvd\" (UID: \"f5b407f8-baf2-467d-817f-ccfc276df9eb\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-58632b-fd88c74d7-rmhvd" Oct 30 01:43:21.381130 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:21.381093 6221 cpu_manager.go:414] "containerMap: removing container" podUID="32567804-08a2-48d7-942d-bd82afff5728" containerName="istio-proxy" Oct 30 01:43:21.381375 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:21.381356 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32567804-08a2-48d7-942d-bd82afff5728" containerName="istio-proxy" Oct 30 01:43:21.391553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:21.386771 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-58632b-78dbfbc54b-857kc"] Oct 30 01:43:21.391553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:21.387198 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-58632b-78dbfbc54b-857kc" Oct 30 01:43:21.440719 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:21.440671 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw54m\" (UniqueName: \"kubernetes.io/projected/f5b407f8-baf2-467d-817f-ccfc276df9eb-kube-api-access-jw54m\") pod \"payload-pre-processing-e2e-praxis-b-58632b-fd88c74d7-rmhvd\" (UID: \"f5b407f8-baf2-467d-817f-ccfc276df9eb\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-58632b-fd88c74d7-rmhvd" Oct 30 01:43:21.440909 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:21.440859 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5b407f8-baf2-467d-817f-ccfc276df9eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-58632b-fd88c74d7-rmhvd\" (UID: \"f5b407f8-baf2-467d-817f-ccfc276df9eb\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-58632b-fd88c74d7-rmhvd" Oct 30 01:43:21.441618 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:21.441599 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5b407f8-baf2-467d-817f-ccfc276df9eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-58632b-fd88c74d7-rmhvd\" (UID: \"f5b407f8-baf2-467d-817f-ccfc276df9eb\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-58632b-fd88c74d7-rmhvd" Oct 30 01:43:21.463624 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:21.463589 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw54m\" (UniqueName: \"kubernetes.io/projected/f5b407f8-baf2-467d-817f-ccfc276df9eb-kube-api-access-jw54m\") pod \"payload-pre-processing-e2e-praxis-b-58632b-fd88c74d7-rmhvd\" (UID: \"f5b407f8-baf2-467d-817f-ccfc276df9eb\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-58632b-fd88c74d7-rmhvd" Oct 30 01:43:21.494637 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:43:21.494596 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:43:21.542613 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:21.542579 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11b32d92-3e3e-48db-a4f5-14b2ed72d7f7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-58632b-78dbfbc54b-857kc\" (UID: \"11b32d92-3e3e-48db-a4f5-14b2ed72d7f7\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-58632b-78dbfbc54b-857kc" Oct 30 01:43:21.542786 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:21.542684 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86npn\" (UniqueName: \"kubernetes.io/projected/11b32d92-3e3e-48db-a4f5-14b2ed72d7f7-kube-api-access-86npn\") pod \"payload-processing-e2e-praxis-b-58632b-78dbfbc54b-857kc\" (UID: \"11b32d92-3e3e-48db-a4f5-14b2ed72d7f7\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-58632b-78dbfbc54b-857kc" Oct 30 01:43:21.661686 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:21.655224 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11b32d92-3e3e-48db-a4f5-14b2ed72d7f7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-58632b-78dbfbc54b-857kc\" (UID: \"11b32d92-3e3e-48db-a4f5-14b2ed72d7f7\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-58632b-78dbfbc54b-857kc" Oct 30 01:43:21.661686 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:21.655357 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86npn\" (UniqueName: \"kubernetes.io/projected/11b32d92-3e3e-48db-a4f5-14b2ed72d7f7-kube-api-access-86npn\") pod \"payload-processing-e2e-praxis-b-58632b-78dbfbc54b-857kc\" (UID: \"11b32d92-3e3e-48db-a4f5-14b2ed72d7f7\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-58632b-78dbfbc54b-857kc" Oct 30 01:43:21.661686 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:21.656675 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11b32d92-3e3e-48db-a4f5-14b2ed72d7f7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-58632b-78dbfbc54b-857kc\" (UID: \"11b32d92-3e3e-48db-a4f5-14b2ed72d7f7\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-58632b-78dbfbc54b-857kc" Oct 30 01:43:21.714555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:21.713708 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86npn\" (UniqueName: \"kubernetes.io/projected/11b32d92-3e3e-48db-a4f5-14b2ed72d7f7-kube-api-access-86npn\") pod \"payload-processing-e2e-praxis-b-58632b-78dbfbc54b-857kc\" (UID: \"11b32d92-3e3e-48db-a4f5-14b2ed72d7f7\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-58632b-78dbfbc54b-857kc" Oct 30 01:43:21.723105 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:21.723044 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-58632b-78dbfbc54b-857kc" Oct 30 01:43:21.759214 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:21.758818 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-58632b-fd88c74d7-rmhvd" Oct 30 01:43:22.086268 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:22.086198 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-58632b-78dbfbc54b-857kc"] Oct 30 01:43:22.089115 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:43:22.089082 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11b32d92_3e3e_48db_a4f5_14b2ed72d7f7.slice/crio-0e2c0edaef9734d1f2166645ae1567f8a97cbd2510a296c2496f2967f4fbe423 WatchSource:0}: Error finding container 0e2c0edaef9734d1f2166645ae1567f8a97cbd2510a296c2496f2967f4fbe423: Status 404 returned error can't find the container with id 0e2c0edaef9734d1f2166645ae1567f8a97cbd2510a296c2496f2967f4fbe423 Oct 30 01:43:22.097293 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:22.097270 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-58632b-fd88c74d7-rmhvd"] Oct 30 01:43:22.105379 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:43:22.105341 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5b407f8_baf2_467d_817f_ccfc276df9eb.slice/crio-0ffa03c0a287f3d50ca88212e0cd92a69747e701d2bed2f08c9c3520ad4d1fed WatchSource:0}: Error finding container 0ffa03c0a287f3d50ca88212e0cd92a69747e701d2bed2f08c9c3520ad4d1fed: Status 404 returned error can't find the container with id 0ffa03c0a287f3d50ca88212e0cd92a69747e701d2bed2f08c9c3520ad4d1fed Oct 30 01:43:22.193049 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:22.193016 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-be8a7d-5dcd7876bd-7h9q5"] Oct 30 01:43:22.197047 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:22.197030 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-be8a7d-5dcd7876bd-7h9q5" Oct 30 01:43:22.213812 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:22.213771 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-be8a7d-serving-cert\"" Oct 30 01:43:22.218344 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:22.218319 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-be8a7d-5dcd7876bd-7h9q5"] Oct 30 01:43:22.373334 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:22.373241 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-be8a7d-5dcd7876bd-7h9q5\" (UID: \"b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-be8a7d-5dcd7876bd-7h9q5" Oct 30 01:43:22.373515 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:22.373369 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw6fj\" (UniqueName: \"kubernetes.io/projected/b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b-kube-api-access-nw6fj\") pod \"maas-api-e2e-praxis-b-be8a7d-5dcd7876bd-7h9q5\" (UID: \"b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-be8a7d-5dcd7876bd-7h9q5" Oct 30 01:43:22.373515 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:22.373444 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b-metrics-certs\") pod \"maas-api-e2e-praxis-b-be8a7d-5dcd7876bd-7h9q5\" (UID: \"b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-be8a7d-5dcd7876bd-7h9q5" Oct 30 01:43:22.373655 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:22.373570 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-be8a7d-5dcd7876bd-7h9q5\" (UID: \"b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-be8a7d-5dcd7876bd-7h9q5" Oct 30 01:43:22.478554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:22.476602 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw6fj\" (UniqueName: \"kubernetes.io/projected/b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b-kube-api-access-nw6fj\") pod \"maas-api-e2e-praxis-b-be8a7d-5dcd7876bd-7h9q5\" (UID: \"b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-be8a7d-5dcd7876bd-7h9q5" Oct 30 01:43:22.478554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:22.477092 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b-metrics-certs\") pod \"maas-api-e2e-praxis-b-be8a7d-5dcd7876bd-7h9q5\" (UID: \"b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-be8a7d-5dcd7876bd-7h9q5" Oct 30 01:43:22.478554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:22.477207 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-be8a7d-5dcd7876bd-7h9q5\" (UID: \"b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-be8a7d-5dcd7876bd-7h9q5" Oct 30 01:43:22.481453 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:22.481366 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-be8a7d-5dcd7876bd-7h9q5\" (UID: \"b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-be8a7d-5dcd7876bd-7h9q5" Oct 30 01:43:22.490620 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:22.477364 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-be8a7d-5dcd7876bd-7h9q5\" (UID: \"b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-be8a7d-5dcd7876bd-7h9q5" Oct 30 01:43:22.491380 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:22.491354 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b-maas-api-tls\") pod \"maas-api-e2e-praxis-b-be8a7d-5dcd7876bd-7h9q5\" (UID: \"b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-be8a7d-5dcd7876bd-7h9q5" Oct 30 01:43:22.514396 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:22.498567 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b-metrics-certs\") pod \"maas-api-e2e-praxis-b-be8a7d-5dcd7876bd-7h9q5\" (UID: \"b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-be8a7d-5dcd7876bd-7h9q5" Oct 30 01:43:22.514396 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:22.510251 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw6fj\" (UniqueName: \"kubernetes.io/projected/b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b-kube-api-access-nw6fj\") pod \"maas-api-e2e-praxis-b-be8a7d-5dcd7876bd-7h9q5\" (UID: \"b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-be8a7d-5dcd7876bd-7h9q5" Oct 30 01:43:22.514788 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:22.511520 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-be8a7d-5dcd7876bd-7h9q5" Oct 30 01:43:22.804889 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:22.804852 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-58632b-fd88c74d7-rmhvd" event={"ID":"f5b407f8-baf2-467d-817f-ccfc276df9eb","Type":"ContainerStarted","Data":"68400605f284eefd5461a3e93e8fcb01f9c6e75b9fda6fce08af1bb980ba11d8"} Oct 30 01:43:22.805390 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:22.804910 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-58632b-fd88c74d7-rmhvd" event={"ID":"f5b407f8-baf2-467d-817f-ccfc276df9eb","Type":"ContainerStarted","Data":"0ffa03c0a287f3d50ca88212e0cd92a69747e701d2bed2f08c9c3520ad4d1fed"} Oct 30 01:43:22.805390 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:22.804940 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-58632b-fd88c74d7-rmhvd" Oct 30 01:43:22.806427 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:22.806403 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-58632b-78dbfbc54b-857kc" event={"ID":"11b32d92-3e3e-48db-a4f5-14b2ed72d7f7","Type":"ContainerStarted","Data":"fa2646bd3c3a283c86cc720f141d0ebef965d17149448aab8a522b749e6281ff"} Oct 30 01:43:22.806575 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:22.806451 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-58632b-78dbfbc54b-857kc" event={"ID":"11b32d92-3e3e-48db-a4f5-14b2ed72d7f7","Type":"ContainerStarted","Data":"0e2c0edaef9734d1f2166645ae1567f8a97cbd2510a296c2496f2967f4fbe423"} Oct 30 01:43:22.806575 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:22.806493 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-58632b-78dbfbc54b-857kc" Oct 30 01:43:22.828825 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:22.828742 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-be8a7d-5dcd7876bd-7h9q5"] Oct 30 01:43:22.831332 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:43:22.831301 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8d53843_bfe3_41a4_92c9_3c7f2ba9eb1b.slice/crio-6f6aad89d86ac035e323e7e18a210eaac38fe4f500e30a1632ab6e6eb2783fb3 WatchSource:0}: Error finding container 6f6aad89d86ac035e323e7e18a210eaac38fe4f500e30a1632ab6e6eb2783fb3: Status 404 returned error can't find the container with id 6f6aad89d86ac035e323e7e18a210eaac38fe4f500e30a1632ab6e6eb2783fb3 Oct 30 01:43:22.832771 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:22.832754 6221 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:43:22.846224 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:22.846177 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-58632b-fd88c74d7-rmhvd" podStartSLOduration=1.846162656 podStartE2EDuration="1.846162656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:43:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:43:22.844793652 +0000 UTC m=+1526.423080989" watchObservedRunningTime="2026-10-30 01:43:22.846162656 +0000 UTC m=+1526.424449991" Oct 30 01:43:22.886060 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:22.885995 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-58632b-78dbfbc54b-857kc" podStartSLOduration=1.8859792880000001 podStartE2EDuration="1.885979288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:43:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:43:22.885022175 +0000 UTC m=+1526.463309512" watchObservedRunningTime="2026-10-30 01:43:22.885979288 +0000 UTC m=+1526.464266623" Oct 30 01:43:23.814178 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:23.814142 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-be8a7d-5dcd7876bd-7h9q5" event={"ID":"b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b","Type":"ContainerStarted","Data":"d2da69f4b3787a0263963a6dc747e2aa3e8bf8547389394a5b9df25654bf7507"} Oct 30 01:43:23.814740 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:23.814385 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-be8a7d-5dcd7876bd-7h9q5" Oct 30 01:43:23.814740 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:23.814438 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-be8a7d-5dcd7876bd-7h9q5" event={"ID":"b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b","Type":"ContainerStarted","Data":"6f6aad89d86ac035e323e7e18a210eaac38fe4f500e30a1632ab6e6eb2783fb3"} Oct 30 01:43:23.854829 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:23.854777 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-be8a7d-5dcd7876bd-7h9q5" podStartSLOduration=1.5183151879999999 podStartE2EDuration="1.854762681s" totalImagesPullingTime="336.447493ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:43:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:43:23.8532692 +0000 UTC m=+1527.431556559" watchObservedRunningTime="2026-10-30 01:43:23.854762681 +0000 UTC m=+1527.433050017" Oct 30 01:43:29.826968 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:29.826939 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-be8a7d-5dcd7876bd-7h9q5" Oct 30 01:43:30.067376 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:30.067338 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-6b3dbd/ipp-route-6b3dbd-kserve-894c568b9-d6fnn"] Oct 30 01:43:33.818197 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:33.818163 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-58632b-fd88c74d7-rmhvd" Oct 30 01:43:33.818613 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:33.818326 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-58632b-78dbfbc54b-857kc" Oct 30 01:43:40.529920 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:40.529836 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3fde1c-6c6f84fb95-4d8dt"] Oct 30 01:43:41.899772 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:41.878702 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q"] Oct 30 01:43:41.899772 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:41.878955 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q" podUID="73b85ea4-7469-459c-8ccd-3e9c669bac1c" containerName="istio-proxy" containerID="cri-o://97db6495da9d4546cbbfc7e92f7595502c258f1d9328e55fe9716867575b14e6" gracePeriod=30 Oct 30 01:43:43.598555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:43.598408 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2ac18f-7bccb6bcbc-qvpb2"] Oct 30 01:43:43.599146 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:43.598669 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2ac18f-7bccb6bcbc-qvpb2" podUID="70f78ce1-27bf-4336-b601-17e028879781" containerName="maas-api" containerID="cri-o://1992f6d5b264d191c0a4d6e9301195829422805c2e1a06bbbe3e7f88fd7d3206" gracePeriod=30 Oct 30 01:43:43.969653 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:43.967799 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2ac18f-9db867f76-qwr6n"] Oct 30 01:43:43.969653 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:43.968477 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-2ac18f-9db867f76-qwr6n" podUID="6f526236-9252-471e-97da-3318a14dd75a" containerName="payload-processing" containerID="cri-o://b0c8f1183419f137a1bafba5288b1839e455e565cbfe862de4949bce791c4d3f" gracePeriod=30 Oct 30 01:43:43.997891 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:43.997855 6221 generic.go:356] "Generic (PLEG): container finished" podID="70f78ce1-27bf-4336-b601-17e028879781" containerID="1992f6d5b264d191c0a4d6e9301195829422805c2e1a06bbbe3e7f88fd7d3206" exitCode=0 Oct 30 01:43:43.998055 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:43.997941 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2ac18f-7bccb6bcbc-qvpb2" event={"ID":"70f78ce1-27bf-4336-b601-17e028879781","Type":"ContainerDied","Data":"1992f6d5b264d191c0a4d6e9301195829422805c2e1a06bbbe3e7f88fd7d3206"} Oct 30 01:43:44.118838 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:44.118526 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2ac18f-7bccb6bcbc-qvpb2" Oct 30 01:43:44.185866 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:44.185836 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70f78ce1-27bf-4336-b601-17e028879781-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70f78ce1-27bf-4336-b601-17e028879781-maas-api-tls\") pod \"70f78ce1-27bf-4336-b601-17e028879781\" (UID: \"70f78ce1-27bf-4336-b601-17e028879781\") " Oct 30 01:43:44.186034 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:44.185999 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70f78ce1-27bf-4336-b601-17e028879781-kube-api-access-mtfpx\" (UniqueName: \"kubernetes.io/projected/70f78ce1-27bf-4336-b601-17e028879781-kube-api-access-mtfpx\") pod \"70f78ce1-27bf-4336-b601-17e028879781\" (UID: \"70f78ce1-27bf-4336-b601-17e028879781\") " Oct 30 01:43:44.186096 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:44.186084 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70f78ce1-27bf-4336-b601-17e028879781-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70f78ce1-27bf-4336-b601-17e028879781-openshift-service-ca\") pod \"70f78ce1-27bf-4336-b601-17e028879781\" (UID: \"70f78ce1-27bf-4336-b601-17e028879781\") " Oct 30 01:43:44.186251 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:44.186195 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70f78ce1-27bf-4336-b601-17e028879781-metrics-certs\" (UniqueName: \"kubernetes.io/secret/70f78ce1-27bf-4336-b601-17e028879781-metrics-certs\") pod \"70f78ce1-27bf-4336-b601-17e028879781\" (UID: \"70f78ce1-27bf-4336-b601-17e028879781\") " Oct 30 01:43:44.186700 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:44.186667 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70f78ce1-27bf-4336-b601-17e028879781-openshift-service-ca" pod "70f78ce1-27bf-4336-b601-17e028879781" (UID: "70f78ce1-27bf-4336-b601-17e028879781"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:43:44.186966 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:44.186932 6221 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70f78ce1-27bf-4336-b601-17e028879781-openshift-service-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:44.189462 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:44.189437 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70f78ce1-27bf-4336-b601-17e028879781-metrics-certs" pod "70f78ce1-27bf-4336-b601-17e028879781" (UID: "70f78ce1-27bf-4336-b601-17e028879781"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:43:44.189587 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:44.189565 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70f78ce1-27bf-4336-b601-17e028879781-maas-api-tls" pod "70f78ce1-27bf-4336-b601-17e028879781" (UID: "70f78ce1-27bf-4336-b601-17e028879781"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:43:44.189751 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:44.189727 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70f78ce1-27bf-4336-b601-17e028879781-kube-api-access-mtfpx" pod "70f78ce1-27bf-4336-b601-17e028879781" (UID: "70f78ce1-27bf-4336-b601-17e028879781"). InnerVolumeSpecName "kube-api-access-mtfpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:43:44.247951 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:44.247922 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2ac18f-9db867f76-qwr6n" Oct 30 01:43:44.373555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:44.330032 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f526236-9252-471e-97da-3318a14dd75a-kube-api-access-dtc4s\" (UniqueName: \"kubernetes.io/projected/6f526236-9252-471e-97da-3318a14dd75a-kube-api-access-dtc4s\") pod \"6f526236-9252-471e-97da-3318a14dd75a\" (UID: \"6f526236-9252-471e-97da-3318a14dd75a\") " Oct 30 01:43:44.373555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:44.330131 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f526236-9252-471e-97da-3318a14dd75a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f526236-9252-471e-97da-3318a14dd75a-plugins-config-volume\") pod \"6f526236-9252-471e-97da-3318a14dd75a\" (UID: \"6f526236-9252-471e-97da-3318a14dd75a\") " Oct 30 01:43:44.373555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:44.330657 6221 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70f78ce1-27bf-4336-b601-17e028879781-metrics-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:44.373555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:44.330681 6221 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70f78ce1-27bf-4336-b601-17e028879781-maas-api-tls\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:44.373555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:44.330695 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtfpx\" (UniqueName: \"kubernetes.io/projected/70f78ce1-27bf-4336-b601-17e028879781-kube-api-access-mtfpx\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:44.412598 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:44.375742 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f526236-9252-471e-97da-3318a14dd75a-kube-api-access-dtc4s" pod "6f526236-9252-471e-97da-3318a14dd75a" (UID: "6f526236-9252-471e-97da-3318a14dd75a"). InnerVolumeSpecName "kube-api-access-dtc4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:43:44.412598 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:44.377769 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f526236-9252-471e-97da-3318a14dd75a-plugins-config-volume" pod "6f526236-9252-471e-97da-3318a14dd75a" (UID: "6f526236-9252-471e-97da-3318a14dd75a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:43:44.461738 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:44.431943 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtc4s\" (UniqueName: \"kubernetes.io/projected/6f526236-9252-471e-97da-3318a14dd75a-kube-api-access-dtc4s\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:44.461738 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:44.431979 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f526236-9252-471e-97da-3318a14dd75a-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:44.570814 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:44.570787 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-58632b-58b9fff455-r2t2s"] Oct 30 01:43:44.571388 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:44.571375 6221 cpu_manager.go:414] "containerMap: removing container" podUID="6f526236-9252-471e-97da-3318a14dd75a" containerName="payload-processing" Oct 30 01:43:44.571431 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:44.571390 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f526236-9252-471e-97da-3318a14dd75a" containerName="payload-processing" Oct 30 01:43:44.571431 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:44.571402 6221 cpu_manager.go:414] "containerMap: removing container" podUID="70f78ce1-27bf-4336-b601-17e028879781" containerName="maas-api" Oct 30 01:43:44.571431 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:44.571408 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70f78ce1-27bf-4336-b601-17e028879781" containerName="maas-api" Oct 30 01:43:44.571525 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:44.571517 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f526236-9252-471e-97da-3318a14dd75a" containerName="payload-processing" Oct 30 01:43:44.571575 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:44.571540 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="70f78ce1-27bf-4336-b601-17e028879781" containerName="maas-api" Oct 30 01:43:44.574990 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:44.574974 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-58632b-58b9fff455-r2t2s" Oct 30 01:43:44.598206 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:44.598169 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-58632b-58b9fff455-r2t2s"] Oct 30 01:43:44.598458 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:44.598437 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-58632b-serving-cert\"" Oct 30 01:43:44.634716 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:44.634672 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f125248b-5ef7-4283-bb8c-44549060fbb8-maas-api-tls\") pod \"maas-api-e2e-praxis-b-58632b-58b9fff455-r2t2s\" (UID: \"f125248b-5ef7-4283-bb8c-44549060fbb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-58632b-58b9fff455-r2t2s" Oct 30 01:43:44.635196 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:44.634724 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f125248b-5ef7-4283-bb8c-44549060fbb8-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-58632b-58b9fff455-r2t2s\" (UID: \"f125248b-5ef7-4283-bb8c-44549060fbb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-58632b-58b9fff455-r2t2s" Oct 30 01:43:44.635196 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:44.634920 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f125248b-5ef7-4283-bb8c-44549060fbb8-metrics-certs\") pod \"maas-api-e2e-praxis-b-58632b-58b9fff455-r2t2s\" (UID: \"f125248b-5ef7-4283-bb8c-44549060fbb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-58632b-58b9fff455-r2t2s" Oct 30 01:43:44.635196 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:44.635136 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4lxn\" (UniqueName: \"kubernetes.io/projected/f125248b-5ef7-4283-bb8c-44549060fbb8-kube-api-access-k4lxn\") pod \"maas-api-e2e-praxis-b-58632b-58b9fff455-r2t2s\" (UID: \"f125248b-5ef7-4283-bb8c-44549060fbb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-58632b-58b9fff455-r2t2s" Oct 30 01:43:44.737029 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:44.736943 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4lxn\" (UniqueName: \"kubernetes.io/projected/f125248b-5ef7-4283-bb8c-44549060fbb8-kube-api-access-k4lxn\") pod \"maas-api-e2e-praxis-b-58632b-58b9fff455-r2t2s\" (UID: \"f125248b-5ef7-4283-bb8c-44549060fbb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-58632b-58b9fff455-r2t2s" Oct 30 01:43:44.737180 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:44.737093 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f125248b-5ef7-4283-bb8c-44549060fbb8-maas-api-tls\") pod \"maas-api-e2e-praxis-b-58632b-58b9fff455-r2t2s\" (UID: \"f125248b-5ef7-4283-bb8c-44549060fbb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-58632b-58b9fff455-r2t2s" Oct 30 01:43:44.737180 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:44.737142 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f125248b-5ef7-4283-bb8c-44549060fbb8-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-58632b-58b9fff455-r2t2s\" (UID: \"f125248b-5ef7-4283-bb8c-44549060fbb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-58632b-58b9fff455-r2t2s" Oct 30 01:43:44.737251 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:44.737201 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f125248b-5ef7-4283-bb8c-44549060fbb8-metrics-certs\") pod \"maas-api-e2e-praxis-b-58632b-58b9fff455-r2t2s\" (UID: \"f125248b-5ef7-4283-bb8c-44549060fbb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-58632b-58b9fff455-r2t2s" Oct 30 01:43:44.737887 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:44.737864 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f125248b-5ef7-4283-bb8c-44549060fbb8-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-58632b-58b9fff455-r2t2s\" (UID: \"f125248b-5ef7-4283-bb8c-44549060fbb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-58632b-58b9fff455-r2t2s" Oct 30 01:43:44.739639 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:44.739618 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f125248b-5ef7-4283-bb8c-44549060fbb8-metrics-certs\") pod \"maas-api-e2e-praxis-b-58632b-58b9fff455-r2t2s\" (UID: \"f125248b-5ef7-4283-bb8c-44549060fbb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-58632b-58b9fff455-r2t2s" Oct 30 01:43:44.739717 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:44.739657 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f125248b-5ef7-4283-bb8c-44549060fbb8-maas-api-tls\") pod \"maas-api-e2e-praxis-b-58632b-58b9fff455-r2t2s\" (UID: \"f125248b-5ef7-4283-bb8c-44549060fbb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-58632b-58b9fff455-r2t2s" Oct 30 01:43:44.757186 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:44.757161 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4lxn\" (UniqueName: \"kubernetes.io/projected/f125248b-5ef7-4283-bb8c-44549060fbb8-kube-api-access-k4lxn\") pod \"maas-api-e2e-praxis-b-58632b-58b9fff455-r2t2s\" (UID: \"f125248b-5ef7-4283-bb8c-44549060fbb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-58632b-58b9fff455-r2t2s" Oct 30 01:43:44.889131 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:44.888767 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-58632b-58b9fff455-r2t2s" Oct 30 01:43:45.006739 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:45.006644 6221 generic.go:356] "Generic (PLEG): container finished" podID="6f526236-9252-471e-97da-3318a14dd75a" containerID="b0c8f1183419f137a1bafba5288b1839e455e565cbfe862de4949bce791c4d3f" exitCode=0 Oct 30 01:43:45.006739 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:45.006692 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2ac18f-9db867f76-qwr6n" Oct 30 01:43:45.006970 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:45.006729 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2ac18f-9db867f76-qwr6n" event={"ID":"6f526236-9252-471e-97da-3318a14dd75a","Type":"ContainerDied","Data":"b0c8f1183419f137a1bafba5288b1839e455e565cbfe862de4949bce791c4d3f"} Oct 30 01:43:45.006970 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:45.006795 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2ac18f-9db867f76-qwr6n" event={"ID":"6f526236-9252-471e-97da-3318a14dd75a","Type":"ContainerDied","Data":"f3b9b8b5d344140c8d306f7a44e1877b00d0674486987e135f8ac2aa05bd4617"} Oct 30 01:43:45.006970 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:45.006828 6221 scope.go:122] "RemoveContainer" containerID="b0c8f1183419f137a1bafba5288b1839e455e565cbfe862de4949bce791c4d3f" Oct 30 01:43:45.008870 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:45.008832 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2ac18f-7bccb6bcbc-qvpb2" event={"ID":"70f78ce1-27bf-4336-b601-17e028879781","Type":"ContainerDied","Data":"383e4c94f00acd66fd172b68bf341f53297f3df0ecc9c099a2ee28288126805a"} Oct 30 01:43:45.009026 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:45.008916 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2ac18f-7bccb6bcbc-qvpb2" Oct 30 01:43:45.018164 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:45.018137 6221 scope.go:122] "RemoveContainer" containerID="b0c8f1183419f137a1bafba5288b1839e455e565cbfe862de4949bce791c4d3f" Oct 30 01:43:45.018464 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:43:45.018441 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0c8f1183419f137a1bafba5288b1839e455e565cbfe862de4949bce791c4d3f\": container with ID starting with b0c8f1183419f137a1bafba5288b1839e455e565cbfe862de4949bce791c4d3f not found: ID does not exist" containerID="b0c8f1183419f137a1bafba5288b1839e455e565cbfe862de4949bce791c4d3f" Oct 30 01:43:45.018594 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:45.018473 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0c8f1183419f137a1bafba5288b1839e455e565cbfe862de4949bce791c4d3f"} err="failed to get container status \"b0c8f1183419f137a1bafba5288b1839e455e565cbfe862de4949bce791c4d3f\": rpc error: code = NotFound desc = could not find container \"b0c8f1183419f137a1bafba5288b1839e455e565cbfe862de4949bce791c4d3f\": container with ID starting with b0c8f1183419f137a1bafba5288b1839e455e565cbfe862de4949bce791c4d3f not found: ID does not exist" Oct 30 01:43:45.018594 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:45.018496 6221 scope.go:122] "RemoveContainer" containerID="1992f6d5b264d191c0a4d6e9301195829422805c2e1a06bbbe3e7f88fd7d3206" Oct 30 01:43:45.060572 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:45.060540 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2ac18f-9db867f76-qwr6n"] Oct 30 01:43:45.066835 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:45.066813 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2ac18f-9db867f76-qwr6n"] Oct 30 01:43:45.088638 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:45.088605 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-6b3dbd/ipp-route-6b3dbd-kserve-894c568b9-d6fnn" podUID="aa2ee40b-7d02-4d12-9894-7d4f51053320" containerName="main" containerID="cri-o://62e967e4891328ea6d73d068750ac60c84669322ee17d8bb832ad3d6b678486f" gracePeriod=45 Oct 30 01:43:45.104467 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:45.104440 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2ac18f-7bccb6bcbc-qvpb2"] Oct 30 01:43:45.110661 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:45.110636 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2ac18f-7bccb6bcbc-qvpb2"] Oct 30 01:43:45.159998 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:45.159959 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-6b3dbd/ipp-route-6b3dbd-kserve-894c568b9-d6fnn" podUID="aa2ee40b-7d02-4d12-9894-7d4f51053320" containerName="main" probeResult="failure" output="Get \"https://10.134.0.113:8000/ready\": dial tcp 10.134.0.113:8000: connect: connection refused" Oct 30 01:43:45.189762 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:45.189726 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-58632b-58b9fff455-r2t2s"] Oct 30 01:43:45.192558 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:43:45.192500 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf125248b_5ef7_4283_bb8c_44549060fbb8.slice/crio-ea909b71d1c767d65b875f0ac10e50fb717cf9f29812681949d7489aecd6b4d4 WatchSource:0}: Error finding container ea909b71d1c767d65b875f0ac10e50fb717cf9f29812681949d7489aecd6b4d4: Status 404 returned error can't find the container with id ea909b71d1c767d65b875f0ac10e50fb717cf9f29812681949d7489aecd6b4d4 Oct 30 01:43:45.338710 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:45.338684 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-6b3dbd/ipp-route-6b3dbd-kserve-894c568b9-d6fnn" Oct 30 01:43:45.450270 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:45.450238 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa2ee40b-7d02-4d12-9894-7d4f51053320-home\" (UniqueName: \"kubernetes.io/empty-dir/aa2ee40b-7d02-4d12-9894-7d4f51053320-home\") pod \"aa2ee40b-7d02-4d12-9894-7d4f51053320\" (UID: \"aa2ee40b-7d02-4d12-9894-7d4f51053320\") " Oct 30 01:43:45.450452 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:45.450308 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa2ee40b-7d02-4d12-9894-7d4f51053320-kube-api-access-nr7nt\" (UniqueName: \"kubernetes.io/projected/aa2ee40b-7d02-4d12-9894-7d4f51053320-kube-api-access-nr7nt\") pod \"aa2ee40b-7d02-4d12-9894-7d4f51053320\" (UID: \"aa2ee40b-7d02-4d12-9894-7d4f51053320\") " Oct 30 01:43:45.450452 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:45.450364 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa2ee40b-7d02-4d12-9894-7d4f51053320-tls-certs\" (UniqueName: \"kubernetes.io/secret/aa2ee40b-7d02-4d12-9894-7d4f51053320-tls-certs\") pod \"aa2ee40b-7d02-4d12-9894-7d4f51053320\" (UID: \"aa2ee40b-7d02-4d12-9894-7d4f51053320\") " Oct 30 01:43:45.450452 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:45.450393 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa2ee40b-7d02-4d12-9894-7d4f51053320-dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa2ee40b-7d02-4d12-9894-7d4f51053320-dshm\") pod \"aa2ee40b-7d02-4d12-9894-7d4f51053320\" (UID: \"aa2ee40b-7d02-4d12-9894-7d4f51053320\") " Oct 30 01:43:45.450655 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:45.450557 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa2ee40b-7d02-4d12-9894-7d4f51053320-home" pod "aa2ee40b-7d02-4d12-9894-7d4f51053320" (UID: "aa2ee40b-7d02-4d12-9894-7d4f51053320"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:43:45.450655 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:45.450581 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa2ee40b-7d02-4d12-9894-7d4f51053320-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa2ee40b-7d02-4d12-9894-7d4f51053320-model-cache\") pod \"aa2ee40b-7d02-4d12-9894-7d4f51053320\" (UID: \"aa2ee40b-7d02-4d12-9894-7d4f51053320\") " Oct 30 01:43:45.450784 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:45.450709 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa2ee40b-7d02-4d12-9894-7d4f51053320-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa2ee40b-7d02-4d12-9894-7d4f51053320-tmp-dir\") pod \"aa2ee40b-7d02-4d12-9894-7d4f51053320\" (UID: \"aa2ee40b-7d02-4d12-9894-7d4f51053320\") " Oct 30 01:43:45.450944 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:45.450917 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa2ee40b-7d02-4d12-9894-7d4f51053320-model-cache" pod "aa2ee40b-7d02-4d12-9894-7d4f51053320" (UID: "aa2ee40b-7d02-4d12-9894-7d4f51053320"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:43:45.451168 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:45.451121 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa2ee40b-7d02-4d12-9894-7d4f51053320-tmp-dir" pod "aa2ee40b-7d02-4d12-9894-7d4f51053320" (UID: "aa2ee40b-7d02-4d12-9894-7d4f51053320"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:43:45.451347 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:45.451327 6221 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa2ee40b-7d02-4d12-9894-7d4f51053320-model-cache\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:45.451421 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:45.451354 6221 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa2ee40b-7d02-4d12-9894-7d4f51053320-tmp-dir\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:45.451421 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:45.451371 6221 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa2ee40b-7d02-4d12-9894-7d4f51053320-home\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:45.452746 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:45.452724 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa2ee40b-7d02-4d12-9894-7d4f51053320-tls-certs" pod "aa2ee40b-7d02-4d12-9894-7d4f51053320" (UID: "aa2ee40b-7d02-4d12-9894-7d4f51053320"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:43:45.452910 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:45.452887 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa2ee40b-7d02-4d12-9894-7d4f51053320-dshm" pod "aa2ee40b-7d02-4d12-9894-7d4f51053320" (UID: "aa2ee40b-7d02-4d12-9894-7d4f51053320"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:43:45.452951 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:45.452906 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa2ee40b-7d02-4d12-9894-7d4f51053320-kube-api-access-nr7nt" pod "aa2ee40b-7d02-4d12-9894-7d4f51053320" (UID: "aa2ee40b-7d02-4d12-9894-7d4f51053320"). InnerVolumeSpecName "kube-api-access-nr7nt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:43:45.534848 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:45.534776 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3fde1c-6c6f84fb95-4d8dt" podUID="f0f39f0a-f1f1-41aa-a9d9-4df2e419ad59" containerName="payload-pre-processing" containerID="cri-o://ea7b368d3ce0389a108e8819cf68a9660565f7ebad1b54ce78062e44e2185bd9" gracePeriod=25 Oct 30 01:43:45.552842 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:45.552748 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr7nt\" (UniqueName: \"kubernetes.io/projected/aa2ee40b-7d02-4d12-9894-7d4f51053320-kube-api-access-nr7nt\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:45.552842 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:45.552776 6221 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa2ee40b-7d02-4d12-9894-7d4f51053320-tls-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:45.552842 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:45.552793 6221 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa2ee40b-7d02-4d12-9894-7d4f51053320-dshm\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:45.805979 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:45.805865 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3fde1c-6c6f84fb95-4d8dt" Oct 30 01:43:45.988808 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:45.988763 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0f39f0a-f1f1-41aa-a9d9-4df2e419ad59-plugins-config-volume" pod "f0f39f0a-f1f1-41aa-a9d9-4df2e419ad59" (UID: "f0f39f0a-f1f1-41aa-a9d9-4df2e419ad59"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:43:46.037475 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:46.035835 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0f39f0a-f1f1-41aa-a9d9-4df2e419ad59-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0f39f0a-f1f1-41aa-a9d9-4df2e419ad59-plugins-config-volume\") pod \"f0f39f0a-f1f1-41aa-a9d9-4df2e419ad59\" (UID: \"f0f39f0a-f1f1-41aa-a9d9-4df2e419ad59\") " Oct 30 01:43:46.037475 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:46.035978 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0f39f0a-f1f1-41aa-a9d9-4df2e419ad59-kube-api-access-klfbv\" (UniqueName: \"kubernetes.io/projected/f0f39f0a-f1f1-41aa-a9d9-4df2e419ad59-kube-api-access-klfbv\") pod \"f0f39f0a-f1f1-41aa-a9d9-4df2e419ad59\" (UID: \"f0f39f0a-f1f1-41aa-a9d9-4df2e419ad59\") " Oct 30 01:43:46.046746 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:46.046709 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0f39f0a-f1f1-41aa-a9d9-4df2e419ad59-kube-api-access-klfbv" pod "f0f39f0a-f1f1-41aa-a9d9-4df2e419ad59" (UID: "f0f39f0a-f1f1-41aa-a9d9-4df2e419ad59"). InnerVolumeSpecName "kube-api-access-klfbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:43:46.047162 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:46.047138 6221 generic.go:356] "Generic (PLEG): container finished" podID="aa2ee40b-7d02-4d12-9894-7d4f51053320" containerID="62e967e4891328ea6d73d068750ac60c84669322ee17d8bb832ad3d6b678486f" exitCode=0 Oct 30 01:43:46.047329 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:46.047314 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-6b3dbd/ipp-route-6b3dbd-kserve-894c568b9-d6fnn" event={"ID":"aa2ee40b-7d02-4d12-9894-7d4f51053320","Type":"ContainerDied","Data":"62e967e4891328ea6d73d068750ac60c84669322ee17d8bb832ad3d6b678486f"} Oct 30 01:43:46.047370 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:46.047347 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-6b3dbd/ipp-route-6b3dbd-kserve-894c568b9-d6fnn" event={"ID":"aa2ee40b-7d02-4d12-9894-7d4f51053320","Type":"ContainerDied","Data":"ca293ce1c0eec6d8b28e93f45223d0d2511bff190ccb507bfd566fae2cbd2869"} Oct 30 01:43:46.047370 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:46.047363 6221 scope.go:122] "RemoveContainer" containerID="62e967e4891328ea6d73d068750ac60c84669322ee17d8bb832ad3d6b678486f" Oct 30 01:43:46.047452 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:46.047443 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-6b3dbd/ipp-route-6b3dbd-kserve-894c568b9-d6fnn" Oct 30 01:43:46.055907 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:46.055884 6221 generic.go:356] "Generic (PLEG): container finished" podID="f0f39f0a-f1f1-41aa-a9d9-4df2e419ad59" containerID="ea7b368d3ce0389a108e8819cf68a9660565f7ebad1b54ce78062e44e2185bd9" exitCode=0 Oct 30 01:43:46.056013 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:46.055960 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3fde1c-6c6f84fb95-4d8dt" event={"ID":"f0f39f0a-f1f1-41aa-a9d9-4df2e419ad59","Type":"ContainerDied","Data":"ea7b368d3ce0389a108e8819cf68a9660565f7ebad1b54ce78062e44e2185bd9"} Oct 30 01:43:46.056013 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:46.055999 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3fde1c-6c6f84fb95-4d8dt" event={"ID":"f0f39f0a-f1f1-41aa-a9d9-4df2e419ad59","Type":"ContainerDied","Data":"988acac4ae18d768679ebaa1644fa1c6335f279c2344a9e658483c59cb3ca9ad"} Oct 30 01:43:46.056114 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:46.056052 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3fde1c-6c6f84fb95-4d8dt" Oct 30 01:43:46.062912 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:46.062882 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-58632b-58b9fff455-r2t2s" event={"ID":"f125248b-5ef7-4283-bb8c-44549060fbb8","Type":"ContainerStarted","Data":"cbf64338b43d6bf513539febcc6632d13138b9f1170bd1f062f8244166c28575"} Oct 30 01:43:46.063022 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:46.062940 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-58632b-58b9fff455-r2t2s" event={"ID":"f125248b-5ef7-4283-bb8c-44549060fbb8","Type":"ContainerStarted","Data":"ea909b71d1c767d65b875f0ac10e50fb717cf9f29812681949d7489aecd6b4d4"} Oct 30 01:43:46.063022 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:46.062975 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-58632b-58b9fff455-r2t2s" Oct 30 01:43:46.066059 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:46.065971 6221 scope.go:122] "RemoveContainer" containerID="62e967e4891328ea6d73d068750ac60c84669322ee17d8bb832ad3d6b678486f" Oct 30 01:43:46.066267 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:43:46.066247 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62e967e4891328ea6d73d068750ac60c84669322ee17d8bb832ad3d6b678486f\": container with ID starting with 62e967e4891328ea6d73d068750ac60c84669322ee17d8bb832ad3d6b678486f not found: ID does not exist" containerID="62e967e4891328ea6d73d068750ac60c84669322ee17d8bb832ad3d6b678486f" Oct 30 01:43:46.066354 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:46.066276 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62e967e4891328ea6d73d068750ac60c84669322ee17d8bb832ad3d6b678486f"} err="failed to get container status \"62e967e4891328ea6d73d068750ac60c84669322ee17d8bb832ad3d6b678486f\": rpc error: code = NotFound desc = could not find container \"62e967e4891328ea6d73d068750ac60c84669322ee17d8bb832ad3d6b678486f\": container with ID starting with 62e967e4891328ea6d73d068750ac60c84669322ee17d8bb832ad3d6b678486f not found: ID does not exist" Oct 30 01:43:46.066354 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:46.066299 6221 scope.go:122] "RemoveContainer" containerID="ea7b368d3ce0389a108e8819cf68a9660565f7ebad1b54ce78062e44e2185bd9" Oct 30 01:43:46.075122 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:46.075104 6221 scope.go:122] "RemoveContainer" containerID="ea7b368d3ce0389a108e8819cf68a9660565f7ebad1b54ce78062e44e2185bd9" Oct 30 01:43:46.075416 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:43:46.075397 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea7b368d3ce0389a108e8819cf68a9660565f7ebad1b54ce78062e44e2185bd9\": container with ID starting with ea7b368d3ce0389a108e8819cf68a9660565f7ebad1b54ce78062e44e2185bd9 not found: ID does not exist" containerID="ea7b368d3ce0389a108e8819cf68a9660565f7ebad1b54ce78062e44e2185bd9" Oct 30 01:43:46.075466 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:46.075422 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea7b368d3ce0389a108e8819cf68a9660565f7ebad1b54ce78062e44e2185bd9"} err="failed to get container status \"ea7b368d3ce0389a108e8819cf68a9660565f7ebad1b54ce78062e44e2185bd9\": rpc error: code = NotFound desc = could not find container \"ea7b368d3ce0389a108e8819cf68a9660565f7ebad1b54ce78062e44e2185bd9\": container with ID starting with ea7b368d3ce0389a108e8819cf68a9660565f7ebad1b54ce78062e44e2185bd9 not found: ID does not exist" Oct 30 01:43:46.108921 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:46.108890 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-6b3dbd/ipp-route-6b3dbd-kserve-894c568b9-d6fnn"] Oct 30 01:43:46.114098 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:46.114072 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-6b3dbd/ipp-route-6b3dbd-kserve-894c568b9-d6fnn"] Oct 30 01:43:46.137779 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:46.137757 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0f39f0a-f1f1-41aa-a9d9-4df2e419ad59-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:46.137779 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:46.137778 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klfbv\" (UniqueName: \"kubernetes.io/projected/f0f39f0a-f1f1-41aa-a9d9-4df2e419ad59-kube-api-access-klfbv\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:46.159355 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:46.159296 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-58632b-58b9fff455-r2t2s" podStartSLOduration=1.8546369600000001 podStartE2EDuration="2.159280096s" totalImagesPullingTime="304.643136ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:43:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:43:46.158634075 +0000 UTC m=+1549.736921411" watchObservedRunningTime="2026-10-30 01:43:46.159280096 +0000 UTC m=+1549.737567432" Oct 30 01:43:46.216304 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:46.212202 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3fde1c-6c6f84fb95-4d8dt"] Oct 30 01:43:46.221792 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:46.221760 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3fde1c-6c6f84fb95-4d8dt"] Oct 30 01:43:46.548593 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:46.548557 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f526236-9252-471e-97da-3318a14dd75a" path="/var/lib/kubelet/pods/6f526236-9252-471e-97da-3318a14dd75a/volumes" Oct 30 01:43:46.549077 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:46.549055 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70f78ce1-27bf-4336-b601-17e028879781" path="/var/lib/kubelet/pods/70f78ce1-27bf-4336-b601-17e028879781/volumes" Oct 30 01:43:46.549568 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:46.549517 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa2ee40b-7d02-4d12-9894-7d4f51053320" path="/var/lib/kubelet/pods/aa2ee40b-7d02-4d12-9894-7d4f51053320/volumes" Oct 30 01:43:46.550283 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:46.550261 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0f39f0a-f1f1-41aa-a9d9-4df2e419ad59" path="/var/lib/kubelet/pods/f0f39f0a-f1f1-41aa-a9d9-4df2e419ad59/volumes" Oct 30 01:43:47.072995 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:47.072961 6221 generic.go:356] "Generic (PLEG): container finished" podID="73b85ea4-7469-459c-8ccd-3e9c669bac1c" containerID="97db6495da9d4546cbbfc7e92f7595502c258f1d9328e55fe9716867575b14e6" exitCode=0 Oct 30 01:43:47.073409 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:47.073042 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q" event={"ID":"73b85ea4-7469-459c-8ccd-3e9c669bac1c","Type":"ContainerDied","Data":"97db6495da9d4546cbbfc7e92f7595502c258f1d9328e55fe9716867575b14e6"} Oct 30 01:43:47.181555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:47.181481 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q" Oct 30 01:43:47.254366 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:47.254331 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73b85ea4-7469-459c-8ccd-3e9c669bac1c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73b85ea4-7469-459c-8ccd-3e9c669bac1c-istio-data\") pod \"73b85ea4-7469-459c-8ccd-3e9c669bac1c\" (UID: \"73b85ea4-7469-459c-8ccd-3e9c669bac1c\") " Oct 30 01:43:47.254599 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:47.254407 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73b85ea4-7469-459c-8ccd-3e9c669bac1c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b85ea4-7469-459c-8ccd-3e9c669bac1c-workload-socket\") pod \"73b85ea4-7469-459c-8ccd-3e9c669bac1c\" (UID: \"73b85ea4-7469-459c-8ccd-3e9c669bac1c\") " Oct 30 01:43:47.254599 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:47.254439 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73b85ea4-7469-459c-8ccd-3e9c669bac1c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73b85ea4-7469-459c-8ccd-3e9c669bac1c-istio-podinfo\") pod \"73b85ea4-7469-459c-8ccd-3e9c669bac1c\" (UID: \"73b85ea4-7469-459c-8ccd-3e9c669bac1c\") " Oct 30 01:43:47.254599 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:47.254540 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73b85ea4-7469-459c-8ccd-3e9c669bac1c-istio-token\" (UniqueName: \"kubernetes.io/projected/73b85ea4-7469-459c-8ccd-3e9c669bac1c-istio-token\") pod \"73b85ea4-7469-459c-8ccd-3e9c669bac1c\" (UID: \"73b85ea4-7469-459c-8ccd-3e9c669bac1c\") " Oct 30 01:43:47.254599 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:47.254588 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73b85ea4-7469-459c-8ccd-3e9c669bac1c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73b85ea4-7469-459c-8ccd-3e9c669bac1c-istiod-ca-cert\") pod \"73b85ea4-7469-459c-8ccd-3e9c669bac1c\" (UID: \"73b85ea4-7469-459c-8ccd-3e9c669bac1c\") " Oct 30 01:43:47.254875 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:47.254617 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73b85ea4-7469-459c-8ccd-3e9c669bac1c-kube-api-access-pr8bk\" (UniqueName: \"kubernetes.io/projected/73b85ea4-7469-459c-8ccd-3e9c669bac1c-kube-api-access-pr8bk\") pod \"73b85ea4-7469-459c-8ccd-3e9c669bac1c\" (UID: \"73b85ea4-7469-459c-8ccd-3e9c669bac1c\") " Oct 30 01:43:47.254875 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:47.254669 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73b85ea4-7469-459c-8ccd-3e9c669bac1c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73b85ea4-7469-459c-8ccd-3e9c669bac1c-workload-certs\") pod \"73b85ea4-7469-459c-8ccd-3e9c669bac1c\" (UID: \"73b85ea4-7469-459c-8ccd-3e9c669bac1c\") " Oct 30 01:43:47.254875 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:47.254675 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73b85ea4-7469-459c-8ccd-3e9c669bac1c-istio-data" pod "73b85ea4-7469-459c-8ccd-3e9c669bac1c" (UID: "73b85ea4-7469-459c-8ccd-3e9c669bac1c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:43:47.254875 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:47.254707 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73b85ea4-7469-459c-8ccd-3e9c669bac1c-workload-socket" pod "73b85ea4-7469-459c-8ccd-3e9c669bac1c" (UID: "73b85ea4-7469-459c-8ccd-3e9c669bac1c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:43:47.254875 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:47.254740 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73b85ea4-7469-459c-8ccd-3e9c669bac1c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b85ea4-7469-459c-8ccd-3e9c669bac1c-credential-socket\") pod \"73b85ea4-7469-459c-8ccd-3e9c669bac1c\" (UID: \"73b85ea4-7469-459c-8ccd-3e9c669bac1c\") " Oct 30 01:43:47.254875 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:47.254778 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73b85ea4-7469-459c-8ccd-3e9c669bac1c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73b85ea4-7469-459c-8ccd-3e9c669bac1c-istio-envoy\") pod \"73b85ea4-7469-459c-8ccd-3e9c669bac1c\" (UID: \"73b85ea4-7469-459c-8ccd-3e9c669bac1c\") " Oct 30 01:43:47.255326 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:47.255294 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73b85ea4-7469-459c-8ccd-3e9c669bac1c-credential-socket" pod "73b85ea4-7469-459c-8ccd-3e9c669bac1c" (UID: "73b85ea4-7469-459c-8ccd-3e9c669bac1c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:43:47.255326 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:47.255317 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73b85ea4-7469-459c-8ccd-3e9c669bac1c-workload-certs" pod "73b85ea4-7469-459c-8ccd-3e9c669bac1c" (UID: "73b85ea4-7469-459c-8ccd-3e9c669bac1c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:43:47.255517 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:47.255418 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b85ea4-7469-459c-8ccd-3e9c669bac1c-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:47.255517 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:47.255432 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73b85ea4-7469-459c-8ccd-3e9c669bac1c-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:47.255517 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:47.255441 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73b85ea4-7469-459c-8ccd-3e9c669bac1c-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:47.255517 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:47.255450 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73b85ea4-7469-459c-8ccd-3e9c669bac1c-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:47.255772 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:47.255716 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73b85ea4-7469-459c-8ccd-3e9c669bac1c-istiod-ca-cert" pod "73b85ea4-7469-459c-8ccd-3e9c669bac1c" (UID: "73b85ea4-7469-459c-8ccd-3e9c669bac1c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:43:47.257986 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:47.257955 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73b85ea4-7469-459c-8ccd-3e9c669bac1c-istio-envoy" pod "73b85ea4-7469-459c-8ccd-3e9c669bac1c" (UID: "73b85ea4-7469-459c-8ccd-3e9c669bac1c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:43:47.258092 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:47.258044 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73b85ea4-7469-459c-8ccd-3e9c669bac1c-istio-podinfo" pod "73b85ea4-7469-459c-8ccd-3e9c669bac1c" (UID: "73b85ea4-7469-459c-8ccd-3e9c669bac1c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:43:47.258193 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:47.258172 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73b85ea4-7469-459c-8ccd-3e9c669bac1c-istio-token" pod "73b85ea4-7469-459c-8ccd-3e9c669bac1c" (UID: "73b85ea4-7469-459c-8ccd-3e9c669bac1c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:43:47.258404 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:47.258380 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73b85ea4-7469-459c-8ccd-3e9c669bac1c-kube-api-access-pr8bk" pod "73b85ea4-7469-459c-8ccd-3e9c669bac1c" (UID: "73b85ea4-7469-459c-8ccd-3e9c669bac1c"). InnerVolumeSpecName "kube-api-access-pr8bk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:43:47.361837 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:47.358441 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73b85ea4-7469-459c-8ccd-3e9c669bac1c-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:47.361837 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:47.358480 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73b85ea4-7469-459c-8ccd-3e9c669bac1c-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:47.361837 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:47.358494 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73b85ea4-7469-459c-8ccd-3e9c669bac1c-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:47.361837 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:47.358509 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pr8bk\" (UniqueName: \"kubernetes.io/projected/73b85ea4-7469-459c-8ccd-3e9c669bac1c-kube-api-access-pr8bk\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:47.361837 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:47.358523 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73b85ea4-7469-459c-8ccd-3e9c669bac1c-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:48.081671 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:48.081633 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q" event={"ID":"73b85ea4-7469-459c-8ccd-3e9c669bac1c","Type":"ContainerDied","Data":"8c2cff409bc319a537c152ba56b8698f2e0f4cfe126408499295ce0ab7076c7e"} Oct 30 01:43:48.081671 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:48.081665 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q" Oct 30 01:43:48.082209 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:48.081701 6221 scope.go:122] "RemoveContainer" containerID="97db6495da9d4546cbbfc7e92f7595502c258f1d9328e55fe9716867575b14e6" Oct 30 01:43:48.140292 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:48.140257 6221 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:43:48.140459 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:48.140313 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q" podUID="73b85ea4-7469-459c-8ccd-3e9c669bac1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:43:48.154071 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:48.154038 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q"] Oct 30 01:43:48.157328 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:48.157308 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-c4z5q"] Oct 30 01:43:48.371950 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:48.371872 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:43:48.547352 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:48.547319 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73b85ea4-7469-459c-8ccd-3e9c669bac1c" path="/var/lib/kubelet/pods/73b85ea4-7469-459c-8ccd-3e9c669bac1c/volumes" Oct 30 01:43:51.494561 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:43:51.494510 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:43:52.080608 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:52.080575 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-58632b-58b9fff455-r2t2s" Oct 30 01:43:52.709424 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:52.709383 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg"] Oct 30 01:43:52.709947 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:52.709682 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" podUID="cda4f86d-d23b-4da5-824e-cacea3b08ab7" containerName="istio-proxy" containerID="cri-o://03e4821dd8e93556f89cf8e286a1b31e0b1e9b96178d11298deeca6d5c88cd46" gracePeriod=30 Oct 30 01:43:57.980636 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:57.980523 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" Oct 30 01:43:58.041065 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:58.041031 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cda4f86d-d23b-4da5-824e-cacea3b08ab7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cda4f86d-d23b-4da5-824e-cacea3b08ab7-istio-envoy\") pod \"cda4f86d-d23b-4da5-824e-cacea3b08ab7\" (UID: \"cda4f86d-d23b-4da5-824e-cacea3b08ab7\") " Oct 30 01:43:58.041256 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:58.041162 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cda4f86d-d23b-4da5-824e-cacea3b08ab7-istio-token\" (UniqueName: \"kubernetes.io/projected/cda4f86d-d23b-4da5-824e-cacea3b08ab7-istio-token\") pod \"cda4f86d-d23b-4da5-824e-cacea3b08ab7\" (UID: \"cda4f86d-d23b-4da5-824e-cacea3b08ab7\") " Oct 30 01:43:58.041256 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:58.041202 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cda4f86d-d23b-4da5-824e-cacea3b08ab7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cda4f86d-d23b-4da5-824e-cacea3b08ab7-credential-socket\") pod \"cda4f86d-d23b-4da5-824e-cacea3b08ab7\" (UID: \"cda4f86d-d23b-4da5-824e-cacea3b08ab7\") " Oct 30 01:43:58.041256 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:58.041251 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cda4f86d-d23b-4da5-824e-cacea3b08ab7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cda4f86d-d23b-4da5-824e-cacea3b08ab7-istio-data\") pod \"cda4f86d-d23b-4da5-824e-cacea3b08ab7\" (UID: \"cda4f86d-d23b-4da5-824e-cacea3b08ab7\") " Oct 30 01:43:58.041449 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:58.041292 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cda4f86d-d23b-4da5-824e-cacea3b08ab7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cda4f86d-d23b-4da5-824e-cacea3b08ab7-istiod-ca-cert\") pod \"cda4f86d-d23b-4da5-824e-cacea3b08ab7\" (UID: \"cda4f86d-d23b-4da5-824e-cacea3b08ab7\") " Oct 30 01:43:58.041449 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:58.041390 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cda4f86d-d23b-4da5-824e-cacea3b08ab7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cda4f86d-d23b-4da5-824e-cacea3b08ab7-workload-certs\") pod \"cda4f86d-d23b-4da5-824e-cacea3b08ab7\" (UID: \"cda4f86d-d23b-4da5-824e-cacea3b08ab7\") " Oct 30 01:43:58.042433 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:58.042401 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cda4f86d-d23b-4da5-824e-cacea3b08ab7-istio-data" pod "cda4f86d-d23b-4da5-824e-cacea3b08ab7" (UID: "cda4f86d-d23b-4da5-824e-cacea3b08ab7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:43:58.042667 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:58.042644 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cda4f86d-d23b-4da5-824e-cacea3b08ab7-credential-socket" pod "cda4f86d-d23b-4da5-824e-cacea3b08ab7" (UID: "cda4f86d-d23b-4da5-824e-cacea3b08ab7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:43:58.042759 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:58.042719 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cda4f86d-d23b-4da5-824e-cacea3b08ab7-istiod-ca-cert" pod "cda4f86d-d23b-4da5-824e-cacea3b08ab7" (UID: "cda4f86d-d23b-4da5-824e-cacea3b08ab7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:43:58.042759 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:58.042734 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cda4f86d-d23b-4da5-824e-cacea3b08ab7-workload-certs" pod "cda4f86d-d23b-4da5-824e-cacea3b08ab7" (UID: "cda4f86d-d23b-4da5-824e-cacea3b08ab7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:43:58.043732 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:58.043705 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cda4f86d-d23b-4da5-824e-cacea3b08ab7-istio-envoy" pod "cda4f86d-d23b-4da5-824e-cacea3b08ab7" (UID: "cda4f86d-d23b-4da5-824e-cacea3b08ab7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:43:58.045035 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:58.045009 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cda4f86d-d23b-4da5-824e-cacea3b08ab7-istio-token" pod "cda4f86d-d23b-4da5-824e-cacea3b08ab7" (UID: "cda4f86d-d23b-4da5-824e-cacea3b08ab7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:43:58.142430 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:58.142395 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cda4f86d-d23b-4da5-824e-cacea3b08ab7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cda4f86d-d23b-4da5-824e-cacea3b08ab7-istio-podinfo\") pod \"cda4f86d-d23b-4da5-824e-cacea3b08ab7\" (UID: \"cda4f86d-d23b-4da5-824e-cacea3b08ab7\") " Oct 30 01:43:58.142641 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:58.142628 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cda4f86d-d23b-4da5-824e-cacea3b08ab7-kube-api-access-lwgns\" (UniqueName: \"kubernetes.io/projected/cda4f86d-d23b-4da5-824e-cacea3b08ab7-kube-api-access-lwgns\") pod \"cda4f86d-d23b-4da5-824e-cacea3b08ab7\" (UID: \"cda4f86d-d23b-4da5-824e-cacea3b08ab7\") " Oct 30 01:43:58.142685 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:58.142666 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cda4f86d-d23b-4da5-824e-cacea3b08ab7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cda4f86d-d23b-4da5-824e-cacea3b08ab7-workload-socket\") pod \"cda4f86d-d23b-4da5-824e-cacea3b08ab7\" (UID: \"cda4f86d-d23b-4da5-824e-cacea3b08ab7\") " Oct 30 01:43:58.142966 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:58.142944 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cda4f86d-d23b-4da5-824e-cacea3b08ab7-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:58.142966 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:58.142947 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cda4f86d-d23b-4da5-824e-cacea3b08ab7-workload-socket" pod "cda4f86d-d23b-4da5-824e-cacea3b08ab7" (UID: "cda4f86d-d23b-4da5-824e-cacea3b08ab7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:43:58.142966 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:58.142965 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cda4f86d-d23b-4da5-824e-cacea3b08ab7-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:58.143218 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:58.142977 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cda4f86d-d23b-4da5-824e-cacea3b08ab7-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:58.143218 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:58.142989 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cda4f86d-d23b-4da5-824e-cacea3b08ab7-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:58.143218 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:58.142997 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cda4f86d-d23b-4da5-824e-cacea3b08ab7-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:58.143218 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:58.143005 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cda4f86d-d23b-4da5-824e-cacea3b08ab7-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:58.144611 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:58.144589 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cda4f86d-d23b-4da5-824e-cacea3b08ab7-istio-podinfo" pod "cda4f86d-d23b-4da5-824e-cacea3b08ab7" (UID: "cda4f86d-d23b-4da5-824e-cacea3b08ab7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:43:58.144719 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:58.144664 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cda4f86d-d23b-4da5-824e-cacea3b08ab7-kube-api-access-lwgns" pod "cda4f86d-d23b-4da5-824e-cacea3b08ab7" (UID: "cda4f86d-d23b-4da5-824e-cacea3b08ab7"). InnerVolumeSpecName "kube-api-access-lwgns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:43:58.163249 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:58.163221 6221 generic.go:356] "Generic (PLEG): container finished" podID="cda4f86d-d23b-4da5-824e-cacea3b08ab7" containerID="03e4821dd8e93556f89cf8e286a1b31e0b1e9b96178d11298deeca6d5c88cd46" exitCode=0 Oct 30 01:43:58.163363 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:58.163282 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" Oct 30 01:43:58.163363 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:58.163304 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" event={"ID":"cda4f86d-d23b-4da5-824e-cacea3b08ab7","Type":"ContainerDied","Data":"03e4821dd8e93556f89cf8e286a1b31e0b1e9b96178d11298deeca6d5c88cd46"} Oct 30 01:43:58.163363 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:58.163357 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg" event={"ID":"cda4f86d-d23b-4da5-824e-cacea3b08ab7","Type":"ContainerDied","Data":"2044cc709ff1cba3d03d79e9d9d29445008906ee79f1398121e9941540399b19"} Oct 30 01:43:58.163478 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:58.163379 6221 scope.go:122] "RemoveContainer" containerID="03e4821dd8e93556f89cf8e286a1b31e0b1e9b96178d11298deeca6d5c88cd46" Oct 30 01:43:58.173926 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:58.173910 6221 scope.go:122] "RemoveContainer" containerID="26843365aab8f7669d8a549e01e9df0149f0c9cccb5e58dae07bc8a3061e6898" Oct 30 01:43:58.184026 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:58.184007 6221 scope.go:122] "RemoveContainer" containerID="03e4821dd8e93556f89cf8e286a1b31e0b1e9b96178d11298deeca6d5c88cd46" Oct 30 01:43:58.184309 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:43:58.184288 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03e4821dd8e93556f89cf8e286a1b31e0b1e9b96178d11298deeca6d5c88cd46\": container with ID starting with 03e4821dd8e93556f89cf8e286a1b31e0b1e9b96178d11298deeca6d5c88cd46 not found: ID does not exist" containerID="03e4821dd8e93556f89cf8e286a1b31e0b1e9b96178d11298deeca6d5c88cd46" Oct 30 01:43:58.184390 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:58.184319 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03e4821dd8e93556f89cf8e286a1b31e0b1e9b96178d11298deeca6d5c88cd46"} err="failed to get container status \"03e4821dd8e93556f89cf8e286a1b31e0b1e9b96178d11298deeca6d5c88cd46\": rpc error: code = NotFound desc = could not find container \"03e4821dd8e93556f89cf8e286a1b31e0b1e9b96178d11298deeca6d5c88cd46\": container with ID starting with 03e4821dd8e93556f89cf8e286a1b31e0b1e9b96178d11298deeca6d5c88cd46 not found: ID does not exist" Oct 30 01:43:58.184390 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:58.184336 6221 scope.go:122] "RemoveContainer" containerID="26843365aab8f7669d8a549e01e9df0149f0c9cccb5e58dae07bc8a3061e6898" Oct 30 01:43:58.184634 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:43:58.184614 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26843365aab8f7669d8a549e01e9df0149f0c9cccb5e58dae07bc8a3061e6898\": container with ID starting with 26843365aab8f7669d8a549e01e9df0149f0c9cccb5e58dae07bc8a3061e6898 not found: ID does not exist" containerID="26843365aab8f7669d8a549e01e9df0149f0c9cccb5e58dae07bc8a3061e6898" Oct 30 01:43:58.184738 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:58.184639 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26843365aab8f7669d8a549e01e9df0149f0c9cccb5e58dae07bc8a3061e6898"} err="failed to get container status \"26843365aab8f7669d8a549e01e9df0149f0c9cccb5e58dae07bc8a3061e6898\": rpc error: code = NotFound desc = could not find container \"26843365aab8f7669d8a549e01e9df0149f0c9cccb5e58dae07bc8a3061e6898\": container with ID starting with 26843365aab8f7669d8a549e01e9df0149f0c9cccb5e58dae07bc8a3061e6898 not found: ID does not exist" Oct 30 01:43:58.221811 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:58.221776 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg"] Oct 30 01:43:58.236222 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:58.236149 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-2ac18f-openshift-default-75b847b9cc-vwjmg"] Oct 30 01:43:58.244690 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:58.244656 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwgns\" (UniqueName: \"kubernetes.io/projected/cda4f86d-d23b-4da5-824e-cacea3b08ab7-kube-api-access-lwgns\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:58.244690 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:58.244691 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cda4f86d-d23b-4da5-824e-cacea3b08ab7-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:58.244903 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:58.244706 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cda4f86d-d23b-4da5-824e-cacea3b08ab7-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:43:58.554745 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:43:58.554707 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cda4f86d-d23b-4da5-824e-cacea3b08ab7" path="/var/lib/kubelet/pods/cda4f86d-d23b-4da5-824e-cacea3b08ab7/volumes" Oct 30 01:44:07.747199 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:07.747160 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c9d174-5f5c577c94-nf2jj"] Oct 30 01:44:07.777585 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:07.777549 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c9d174-5479cc575f-lc7f8"] Oct 30 01:44:12.748133 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:12.748059 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-c9d174-5f5c577c94-nf2jj" podUID="597150a2-dcbe-4a84-9a68-41eb187cbce4" containerName="payload-processing" containerID="cri-o://bbae8d8d4952cbe06953bb10aa0e3775cb566b6aefcedd23be743209df9fa86a" gracePeriod=25 Oct 30 01:44:12.782651 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:12.778118 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c9d174-5479cc575f-lc7f8" podUID="cec66c15-dfbf-4cc8-b4e7-bc42bdc45bae" containerName="payload-pre-processing" containerID="cri-o://527abbadaf9a735be77b18bc5515319a1b9e8a41a05209fd9d6771f344a41ec1" gracePeriod=25 Oct 30 01:44:13.085519 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:13.085491 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c9d174-5f5c577c94-nf2jj" Oct 30 01:44:13.112991 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:13.112965 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c9d174-5479cc575f-lc7f8" Oct 30 01:44:13.134972 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:13.133736 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/597150a2-dcbe-4a84-9a68-41eb187cbce4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/597150a2-dcbe-4a84-9a68-41eb187cbce4-plugins-config-volume\") pod \"597150a2-dcbe-4a84-9a68-41eb187cbce4\" (UID: \"597150a2-dcbe-4a84-9a68-41eb187cbce4\") " Oct 30 01:44:13.134972 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:13.133794 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/597150a2-dcbe-4a84-9a68-41eb187cbce4-kube-api-access-ht4lz\" (UniqueName: \"kubernetes.io/projected/597150a2-dcbe-4a84-9a68-41eb187cbce4-kube-api-access-ht4lz\") pod \"597150a2-dcbe-4a84-9a68-41eb187cbce4\" (UID: \"597150a2-dcbe-4a84-9a68-41eb187cbce4\") " Oct 30 01:44:13.134972 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:13.133984 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cec66c15-dfbf-4cc8-b4e7-bc42bdc45bae-kube-api-access-s8cb6\" (UniqueName: \"kubernetes.io/projected/cec66c15-dfbf-4cc8-b4e7-bc42bdc45bae-kube-api-access-s8cb6\") pod \"cec66c15-dfbf-4cc8-b4e7-bc42bdc45bae\" (UID: \"cec66c15-dfbf-4cc8-b4e7-bc42bdc45bae\") " Oct 30 01:44:13.134972 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:13.134100 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cec66c15-dfbf-4cc8-b4e7-bc42bdc45bae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cec66c15-dfbf-4cc8-b4e7-bc42bdc45bae-plugins-config-volume\") pod \"cec66c15-dfbf-4cc8-b4e7-bc42bdc45bae\" (UID: \"cec66c15-dfbf-4cc8-b4e7-bc42bdc45bae\") " Oct 30 01:44:13.134972 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:13.134417 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/597150a2-dcbe-4a84-9a68-41eb187cbce4-plugins-config-volume" pod "597150a2-dcbe-4a84-9a68-41eb187cbce4" (UID: "597150a2-dcbe-4a84-9a68-41eb187cbce4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:44:13.134972 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:13.134834 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/597150a2-dcbe-4a84-9a68-41eb187cbce4-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:44:13.135740 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:13.135701 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cec66c15-dfbf-4cc8-b4e7-bc42bdc45bae-plugins-config-volume" pod "cec66c15-dfbf-4cc8-b4e7-bc42bdc45bae" (UID: "cec66c15-dfbf-4cc8-b4e7-bc42bdc45bae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:44:13.137210 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:13.137185 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/597150a2-dcbe-4a84-9a68-41eb187cbce4-kube-api-access-ht4lz" pod "597150a2-dcbe-4a84-9a68-41eb187cbce4" (UID: "597150a2-dcbe-4a84-9a68-41eb187cbce4"). InnerVolumeSpecName "kube-api-access-ht4lz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:44:13.138138 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:13.138116 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cec66c15-dfbf-4cc8-b4e7-bc42bdc45bae-kube-api-access-s8cb6" pod "cec66c15-dfbf-4cc8-b4e7-bc42bdc45bae" (UID: "cec66c15-dfbf-4cc8-b4e7-bc42bdc45bae"). InnerVolumeSpecName "kube-api-access-s8cb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:44:13.236700 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:13.236667 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8cb6\" (UniqueName: \"kubernetes.io/projected/cec66c15-dfbf-4cc8-b4e7-bc42bdc45bae-kube-api-access-s8cb6\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:44:13.236700 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:13.236698 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cec66c15-dfbf-4cc8-b4e7-bc42bdc45bae-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:44:13.236908 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:13.236710 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ht4lz\" (UniqueName: \"kubernetes.io/projected/597150a2-dcbe-4a84-9a68-41eb187cbce4-kube-api-access-ht4lz\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:44:13.274789 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:13.274753 6221 generic.go:356] "Generic (PLEG): container finished" podID="cec66c15-dfbf-4cc8-b4e7-bc42bdc45bae" containerID="527abbadaf9a735be77b18bc5515319a1b9e8a41a05209fd9d6771f344a41ec1" exitCode=0 Oct 30 01:44:13.274970 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:13.274799 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c9d174-5479cc575f-lc7f8" Oct 30 01:44:13.274970 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:13.274843 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c9d174-5479cc575f-lc7f8" event={"ID":"cec66c15-dfbf-4cc8-b4e7-bc42bdc45bae","Type":"ContainerDied","Data":"527abbadaf9a735be77b18bc5515319a1b9e8a41a05209fd9d6771f344a41ec1"} Oct 30 01:44:13.274970 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:13.274902 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c9d174-5479cc575f-lc7f8" event={"ID":"cec66c15-dfbf-4cc8-b4e7-bc42bdc45bae","Type":"ContainerDied","Data":"18e58bdd448ddbc941c445eb6abc1cca28c1b1fd60e0d15e0899bbf36637eeae"} Oct 30 01:44:13.274970 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:13.274931 6221 scope.go:122] "RemoveContainer" containerID="527abbadaf9a735be77b18bc5515319a1b9e8a41a05209fd9d6771f344a41ec1" Oct 30 01:44:13.276329 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:13.276305 6221 generic.go:356] "Generic (PLEG): container finished" podID="597150a2-dcbe-4a84-9a68-41eb187cbce4" containerID="bbae8d8d4952cbe06953bb10aa0e3775cb566b6aefcedd23be743209df9fa86a" exitCode=0 Oct 30 01:44:13.276455 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:13.276372 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c9d174-5f5c577c94-nf2jj" Oct 30 01:44:13.276455 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:13.276412 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c9d174-5f5c577c94-nf2jj" event={"ID":"597150a2-dcbe-4a84-9a68-41eb187cbce4","Type":"ContainerDied","Data":"bbae8d8d4952cbe06953bb10aa0e3775cb566b6aefcedd23be743209df9fa86a"} Oct 30 01:44:13.276593 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:13.276461 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c9d174-5f5c577c94-nf2jj" event={"ID":"597150a2-dcbe-4a84-9a68-41eb187cbce4","Type":"ContainerDied","Data":"e898fbad79593d800eeac0399f6810c707ae08c2159a35d81a7211c2933a988f"} Oct 30 01:44:13.291123 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:13.291104 6221 scope.go:122] "RemoveContainer" containerID="527abbadaf9a735be77b18bc5515319a1b9e8a41a05209fd9d6771f344a41ec1" Oct 30 01:44:13.291578 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:44:13.291513 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"527abbadaf9a735be77b18bc5515319a1b9e8a41a05209fd9d6771f344a41ec1\": container with ID starting with 527abbadaf9a735be77b18bc5515319a1b9e8a41a05209fd9d6771f344a41ec1 not found: ID does not exist" containerID="527abbadaf9a735be77b18bc5515319a1b9e8a41a05209fd9d6771f344a41ec1" Oct 30 01:44:13.291671 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:13.291589 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"527abbadaf9a735be77b18bc5515319a1b9e8a41a05209fd9d6771f344a41ec1"} err="failed to get container status \"527abbadaf9a735be77b18bc5515319a1b9e8a41a05209fd9d6771f344a41ec1\": rpc error: code = NotFound desc = could not find container \"527abbadaf9a735be77b18bc5515319a1b9e8a41a05209fd9d6771f344a41ec1\": container with ID starting with 527abbadaf9a735be77b18bc5515319a1b9e8a41a05209fd9d6771f344a41ec1 not found: ID does not exist" Oct 30 01:44:13.291671 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:13.291615 6221 scope.go:122] "RemoveContainer" containerID="bbae8d8d4952cbe06953bb10aa0e3775cb566b6aefcedd23be743209df9fa86a" Oct 30 01:44:13.300982 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:13.300964 6221 scope.go:122] "RemoveContainer" containerID="bbae8d8d4952cbe06953bb10aa0e3775cb566b6aefcedd23be743209df9fa86a" Oct 30 01:44:13.301284 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:44:13.301267 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbae8d8d4952cbe06953bb10aa0e3775cb566b6aefcedd23be743209df9fa86a\": container with ID starting with bbae8d8d4952cbe06953bb10aa0e3775cb566b6aefcedd23be743209df9fa86a not found: ID does not exist" containerID="bbae8d8d4952cbe06953bb10aa0e3775cb566b6aefcedd23be743209df9fa86a" Oct 30 01:44:13.301339 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:13.301291 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbae8d8d4952cbe06953bb10aa0e3775cb566b6aefcedd23be743209df9fa86a"} err="failed to get container status \"bbae8d8d4952cbe06953bb10aa0e3775cb566b6aefcedd23be743209df9fa86a\": rpc error: code = NotFound desc = could not find container \"bbae8d8d4952cbe06953bb10aa0e3775cb566b6aefcedd23be743209df9fa86a\": container with ID starting with bbae8d8d4952cbe06953bb10aa0e3775cb566b6aefcedd23be743209df9fa86a not found: ID does not exist" Oct 30 01:44:13.342828 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:13.342800 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c9d174-5479cc575f-lc7f8"] Oct 30 01:44:13.347164 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:13.347136 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c9d174-5479cc575f-lc7f8"] Oct 30 01:44:13.379624 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:13.379590 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c9d174-5f5c577c94-nf2jj"] Oct 30 01:44:13.384730 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:13.384701 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c9d174-5f5c577c94-nf2jj"] Oct 30 01:44:14.547513 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:14.547478 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="597150a2-dcbe-4a84-9a68-41eb187cbce4" path="/var/lib/kubelet/pods/597150a2-dcbe-4a84-9a68-41eb187cbce4/volumes" Oct 30 01:44:14.547960 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:14.547943 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cec66c15-dfbf-4cc8-b4e7-bc42bdc45bae" path="/var/lib/kubelet/pods/cec66c15-dfbf-4cc8-b4e7-bc42bdc45bae/volumes" Oct 30 01:44:21.494217 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:44:21.494178 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:44:27.336836 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:27.336805 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f01cbc-5594bbb79d-bhxg9"] Oct 30 01:44:27.337494 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:27.337039 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f01cbc-5594bbb79d-bhxg9" podUID="f2416576-64de-4ee6-8082-52b1c1bb5e91" containerName="maas-api" containerID="cri-o://44a9f54723cbb45bc882256a1ab86218a8e2220efc886d97d7039314bb001e1c" gracePeriod=30 Oct 30 01:44:27.523475 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:27.523441 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f01cbc-77449896b-mzbkb"] Oct 30 01:44:27.523765 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:27.523735 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f01cbc-77449896b-mzbkb" podUID="74c960b2-ff58-455b-8eb4-43cc4171e32c" containerName="payload-processing" containerID="cri-o://40e42cc878a25ff472659ca140970cd03db607fc5f7620e22d7309d8130d5758" gracePeriod=30 Oct 30 01:44:27.567718 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:27.567684 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f01cbc-7f6fdd8bb7-pkht9"] Oct 30 01:44:27.567923 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:27.567901 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f01cbc-7f6fdd8bb7-pkht9" podUID="8f2b1f15-868f-4126-bf67-b65a2e5b50a7" containerName="payload-pre-processing" containerID="cri-o://5b1475a1959ebbf2478c733b97d39e4c65275bd63ebbd71fb271e288dee05961" gracePeriod=30 Oct 30 01:44:27.693630 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:27.693600 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f01cbc-5594bbb79d-bhxg9" Oct 30 01:44:27.779205 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:27.778794 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2416576-64de-4ee6-8082-52b1c1bb5e91-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2416576-64de-4ee6-8082-52b1c1bb5e91-openshift-service-ca\") pod \"f2416576-64de-4ee6-8082-52b1c1bb5e91\" (UID: \"f2416576-64de-4ee6-8082-52b1c1bb5e91\") " Oct 30 01:44:27.779205 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:27.778973 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2416576-64de-4ee6-8082-52b1c1bb5e91-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2416576-64de-4ee6-8082-52b1c1bb5e91-metrics-certs\") pod \"f2416576-64de-4ee6-8082-52b1c1bb5e91\" (UID: \"f2416576-64de-4ee6-8082-52b1c1bb5e91\") " Oct 30 01:44:27.779205 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:27.779083 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2416576-64de-4ee6-8082-52b1c1bb5e91-kube-api-access-rp4ls\" (UniqueName: \"kubernetes.io/projected/f2416576-64de-4ee6-8082-52b1c1bb5e91-kube-api-access-rp4ls\") pod \"f2416576-64de-4ee6-8082-52b1c1bb5e91\" (UID: \"f2416576-64de-4ee6-8082-52b1c1bb5e91\") " Oct 30 01:44:27.779205 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:27.779165 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2416576-64de-4ee6-8082-52b1c1bb5e91-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2416576-64de-4ee6-8082-52b1c1bb5e91-maas-api-tls\") pod \"f2416576-64de-4ee6-8082-52b1c1bb5e91\" (UID: \"f2416576-64de-4ee6-8082-52b1c1bb5e91\") " Oct 30 01:44:27.780858 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:27.780472 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2416576-64de-4ee6-8082-52b1c1bb5e91-openshift-service-ca" pod "f2416576-64de-4ee6-8082-52b1c1bb5e91" (UID: "f2416576-64de-4ee6-8082-52b1c1bb5e91"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:44:27.783120 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:27.783032 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2416576-64de-4ee6-8082-52b1c1bb5e91-maas-api-tls" pod "f2416576-64de-4ee6-8082-52b1c1bb5e91" (UID: "f2416576-64de-4ee6-8082-52b1c1bb5e91"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:44:27.784296 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:27.784264 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2416576-64de-4ee6-8082-52b1c1bb5e91-metrics-certs" pod "f2416576-64de-4ee6-8082-52b1c1bb5e91" (UID: "f2416576-64de-4ee6-8082-52b1c1bb5e91"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:44:27.787054 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:27.787018 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2416576-64de-4ee6-8082-52b1c1bb5e91-kube-api-access-rp4ls" pod "f2416576-64de-4ee6-8082-52b1c1bb5e91" (UID: "f2416576-64de-4ee6-8082-52b1c1bb5e91"). InnerVolumeSpecName "kube-api-access-rp4ls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:44:27.820125 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:27.820096 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f01cbc-77449896b-mzbkb" Oct 30 01:44:27.881645 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:27.880336 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74c960b2-ff58-455b-8eb4-43cc4171e32c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74c960b2-ff58-455b-8eb4-43cc4171e32c-plugins-config-volume\") pod \"74c960b2-ff58-455b-8eb4-43cc4171e32c\" (UID: \"74c960b2-ff58-455b-8eb4-43cc4171e32c\") " Oct 30 01:44:27.881645 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:27.880493 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74c960b2-ff58-455b-8eb4-43cc4171e32c-kube-api-access-bmrqf\" (UniqueName: \"kubernetes.io/projected/74c960b2-ff58-455b-8eb4-43cc4171e32c-kube-api-access-bmrqf\") pod \"74c960b2-ff58-455b-8eb4-43cc4171e32c\" (UID: \"74c960b2-ff58-455b-8eb4-43cc4171e32c\") " Oct 30 01:44:27.881645 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:27.880999 6221 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2416576-64de-4ee6-8082-52b1c1bb5e91-maas-api-tls\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:44:27.881645 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:27.881023 6221 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2416576-64de-4ee6-8082-52b1c1bb5e91-openshift-service-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:44:27.881645 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:27.881037 6221 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2416576-64de-4ee6-8082-52b1c1bb5e91-metrics-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:44:27.881645 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:27.881050 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp4ls\" (UniqueName: \"kubernetes.io/projected/f2416576-64de-4ee6-8082-52b1c1bb5e91-kube-api-access-rp4ls\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:44:27.895561 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:27.893181 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74c960b2-ff58-455b-8eb4-43cc4171e32c-plugins-config-volume" pod "74c960b2-ff58-455b-8eb4-43cc4171e32c" (UID: "74c960b2-ff58-455b-8eb4-43cc4171e32c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:44:27.918553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:27.917844 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74c960b2-ff58-455b-8eb4-43cc4171e32c-kube-api-access-bmrqf" pod "74c960b2-ff58-455b-8eb4-43cc4171e32c" (UID: "74c960b2-ff58-455b-8eb4-43cc4171e32c"). InnerVolumeSpecName "kube-api-access-bmrqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:44:27.985574 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:27.982059 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/74c960b2-ff58-455b-8eb4-43cc4171e32c-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:44:27.985574 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:27.982086 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmrqf\" (UniqueName: \"kubernetes.io/projected/74c960b2-ff58-455b-8eb4-43cc4171e32c-kube-api-access-bmrqf\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:44:28.017856 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:28.017823 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f01cbc-7f6fdd8bb7-pkht9" Oct 30 01:44:28.082707 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:28.082668 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f2b1f15-868f-4126-bf67-b65a2e5b50a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f2b1f15-868f-4126-bf67-b65a2e5b50a7-plugins-config-volume\") pod \"8f2b1f15-868f-4126-bf67-b65a2e5b50a7\" (UID: \"8f2b1f15-868f-4126-bf67-b65a2e5b50a7\") " Oct 30 01:44:28.082897 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:28.082786 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f2b1f15-868f-4126-bf67-b65a2e5b50a7-kube-api-access-dqj7c\" (UniqueName: \"kubernetes.io/projected/8f2b1f15-868f-4126-bf67-b65a2e5b50a7-kube-api-access-dqj7c\") pod \"8f2b1f15-868f-4126-bf67-b65a2e5b50a7\" (UID: \"8f2b1f15-868f-4126-bf67-b65a2e5b50a7\") " Oct 30 01:44:28.083074 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:28.083052 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f2b1f15-868f-4126-bf67-b65a2e5b50a7-plugins-config-volume" pod "8f2b1f15-868f-4126-bf67-b65a2e5b50a7" (UID: "8f2b1f15-868f-4126-bf67-b65a2e5b50a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:44:28.083273 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:28.083257 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f2b1f15-868f-4126-bf67-b65a2e5b50a7-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:44:28.084929 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:28.084907 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f2b1f15-868f-4126-bf67-b65a2e5b50a7-kube-api-access-dqj7c" pod "8f2b1f15-868f-4126-bf67-b65a2e5b50a7" (UID: "8f2b1f15-868f-4126-bf67-b65a2e5b50a7"). InnerVolumeSpecName "kube-api-access-dqj7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:44:28.185208 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:28.185134 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqj7c\" (UniqueName: \"kubernetes.io/projected/8f2b1f15-868f-4126-bf67-b65a2e5b50a7-kube-api-access-dqj7c\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:44:28.407845 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:28.407809 6221 generic.go:356] "Generic (PLEG): container finished" podID="8f2b1f15-868f-4126-bf67-b65a2e5b50a7" containerID="5b1475a1959ebbf2478c733b97d39e4c65275bd63ebbd71fb271e288dee05961" exitCode=0 Oct 30 01:44:28.408275 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:28.407857 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f01cbc-7f6fdd8bb7-pkht9" Oct 30 01:44:28.408275 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:28.407893 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f01cbc-7f6fdd8bb7-pkht9" event={"ID":"8f2b1f15-868f-4126-bf67-b65a2e5b50a7","Type":"ContainerDied","Data":"5b1475a1959ebbf2478c733b97d39e4c65275bd63ebbd71fb271e288dee05961"} Oct 30 01:44:28.408275 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:28.407956 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f01cbc-7f6fdd8bb7-pkht9" event={"ID":"8f2b1f15-868f-4126-bf67-b65a2e5b50a7","Type":"ContainerDied","Data":"25b7e26051df30ea072ee4d64eb041ea1b4003723456673c435b98ec21b85f76"} Oct 30 01:44:28.408275 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:28.407988 6221 scope.go:122] "RemoveContainer" containerID="5b1475a1959ebbf2478c733b97d39e4c65275bd63ebbd71fb271e288dee05961" Oct 30 01:44:28.409355 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:28.409242 6221 generic.go:356] "Generic (PLEG): container finished" podID="74c960b2-ff58-455b-8eb4-43cc4171e32c" containerID="40e42cc878a25ff472659ca140970cd03db607fc5f7620e22d7309d8130d5758" exitCode=0 Oct 30 01:44:28.409355 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:28.409282 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f01cbc-77449896b-mzbkb" Oct 30 01:44:28.409355 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:28.409325 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f01cbc-77449896b-mzbkb" event={"ID":"74c960b2-ff58-455b-8eb4-43cc4171e32c","Type":"ContainerDied","Data":"40e42cc878a25ff472659ca140970cd03db607fc5f7620e22d7309d8130d5758"} Oct 30 01:44:28.409666 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:28.409374 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f01cbc-77449896b-mzbkb" event={"ID":"74c960b2-ff58-455b-8eb4-43cc4171e32c","Type":"ContainerDied","Data":"d63c7520799ba30f4aca5452bb55134e38da07bec21dcf9587cbcb2c9d1dbfe9"} Oct 30 01:44:28.411065 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:28.411041 6221 generic.go:356] "Generic (PLEG): container finished" podID="f2416576-64de-4ee6-8082-52b1c1bb5e91" containerID="44a9f54723cbb45bc882256a1ab86218a8e2220efc886d97d7039314bb001e1c" exitCode=0 Oct 30 01:44:28.411191 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:28.411134 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f01cbc-5594bbb79d-bhxg9" event={"ID":"f2416576-64de-4ee6-8082-52b1c1bb5e91","Type":"ContainerDied","Data":"44a9f54723cbb45bc882256a1ab86218a8e2220efc886d97d7039314bb001e1c"} Oct 30 01:44:28.411191 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:28.411166 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f01cbc-5594bbb79d-bhxg9" event={"ID":"f2416576-64de-4ee6-8082-52b1c1bb5e91","Type":"ContainerDied","Data":"2d277cf61d878b0d37c8357f93bc1194e9abf2bdbca63cb9f1028e2a6d377b2d"} Oct 30 01:44:28.411333 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:28.411215 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f01cbc-5594bbb79d-bhxg9" Oct 30 01:44:28.418505 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:28.418488 6221 scope.go:122] "RemoveContainer" containerID="5b1475a1959ebbf2478c733b97d39e4c65275bd63ebbd71fb271e288dee05961" Oct 30 01:44:28.419767 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:44:28.419280 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b1475a1959ebbf2478c733b97d39e4c65275bd63ebbd71fb271e288dee05961\": container with ID starting with 5b1475a1959ebbf2478c733b97d39e4c65275bd63ebbd71fb271e288dee05961 not found: ID does not exist" containerID="5b1475a1959ebbf2478c733b97d39e4c65275bd63ebbd71fb271e288dee05961" Oct 30 01:44:28.419767 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:28.419317 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b1475a1959ebbf2478c733b97d39e4c65275bd63ebbd71fb271e288dee05961"} err="failed to get container status \"5b1475a1959ebbf2478c733b97d39e4c65275bd63ebbd71fb271e288dee05961\": rpc error: code = NotFound desc = could not find container \"5b1475a1959ebbf2478c733b97d39e4c65275bd63ebbd71fb271e288dee05961\": container with ID starting with 5b1475a1959ebbf2478c733b97d39e4c65275bd63ebbd71fb271e288dee05961 not found: ID does not exist" Oct 30 01:44:28.419767 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:28.419342 6221 scope.go:122] "RemoveContainer" containerID="40e42cc878a25ff472659ca140970cd03db607fc5f7620e22d7309d8130d5758" Oct 30 01:44:28.429468 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:28.429445 6221 scope.go:122] "RemoveContainer" containerID="40e42cc878a25ff472659ca140970cd03db607fc5f7620e22d7309d8130d5758" Oct 30 01:44:28.429753 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:44:28.429733 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40e42cc878a25ff472659ca140970cd03db607fc5f7620e22d7309d8130d5758\": container with ID starting with 40e42cc878a25ff472659ca140970cd03db607fc5f7620e22d7309d8130d5758 not found: ID does not exist" containerID="40e42cc878a25ff472659ca140970cd03db607fc5f7620e22d7309d8130d5758" Oct 30 01:44:28.429872 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:28.429760 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40e42cc878a25ff472659ca140970cd03db607fc5f7620e22d7309d8130d5758"} err="failed to get container status \"40e42cc878a25ff472659ca140970cd03db607fc5f7620e22d7309d8130d5758\": rpc error: code = NotFound desc = could not find container \"40e42cc878a25ff472659ca140970cd03db607fc5f7620e22d7309d8130d5758\": container with ID starting with 40e42cc878a25ff472659ca140970cd03db607fc5f7620e22d7309d8130d5758 not found: ID does not exist" Oct 30 01:44:28.429872 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:28.429776 6221 scope.go:122] "RemoveContainer" containerID="44a9f54723cbb45bc882256a1ab86218a8e2220efc886d97d7039314bb001e1c" Oct 30 01:44:28.439118 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:28.439099 6221 scope.go:122] "RemoveContainer" containerID="44a9f54723cbb45bc882256a1ab86218a8e2220efc886d97d7039314bb001e1c" Oct 30 01:44:28.439423 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:44:28.439404 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44a9f54723cbb45bc882256a1ab86218a8e2220efc886d97d7039314bb001e1c\": container with ID starting with 44a9f54723cbb45bc882256a1ab86218a8e2220efc886d97d7039314bb001e1c not found: ID does not exist" containerID="44a9f54723cbb45bc882256a1ab86218a8e2220efc886d97d7039314bb001e1c" Oct 30 01:44:28.439475 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:28.439431 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44a9f54723cbb45bc882256a1ab86218a8e2220efc886d97d7039314bb001e1c"} err="failed to get container status \"44a9f54723cbb45bc882256a1ab86218a8e2220efc886d97d7039314bb001e1c\": rpc error: code = NotFound desc = could not find container \"44a9f54723cbb45bc882256a1ab86218a8e2220efc886d97d7039314bb001e1c\": container with ID starting with 44a9f54723cbb45bc882256a1ab86218a8e2220efc886d97d7039314bb001e1c not found: ID does not exist" Oct 30 01:44:28.496743 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:28.496716 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f01cbc-7f6fdd8bb7-pkht9"] Oct 30 01:44:28.505573 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:28.505548 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f01cbc-7f6fdd8bb7-pkht9"] Oct 30 01:44:28.549169 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:28.549126 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f2b1f15-868f-4126-bf67-b65a2e5b50a7" path="/var/lib/kubelet/pods/8f2b1f15-868f-4126-bf67-b65a2e5b50a7/volumes" Oct 30 01:44:28.559474 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:28.559434 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f01cbc-5594bbb79d-bhxg9"] Oct 30 01:44:28.562074 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:28.562054 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f01cbc-5594bbb79d-bhxg9"] Oct 30 01:44:28.593616 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:28.593581 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f01cbc-77449896b-mzbkb"] Oct 30 01:44:28.597927 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:28.597901 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f01cbc-77449896b-mzbkb"] Oct 30 01:44:29.148219 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:29.148189 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:44:30.584554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.571163 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74c960b2-ff58-455b-8eb4-43cc4171e32c" path="/var/lib/kubelet/pods/74c960b2-ff58-455b-8eb4-43cc4171e32c/volumes" Oct 30 01:44:30.584554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.571890 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2416576-64de-4ee6-8082-52b1c1bb5e91" path="/var/lib/kubelet/pods/f2416576-64de-4ee6-8082-52b1c1bb5e91/volumes" Oct 30 01:44:30.627311 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.627268 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-0ae258/praxis-route-0ae258-kserve-fb65bffff-s8cc2"] Oct 30 01:44:30.629469 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.627912 6221 cpu_manager.go:414] "containerMap: removing container" podUID="597150a2-dcbe-4a84-9a68-41eb187cbce4" containerName="payload-processing" Oct 30 01:44:30.631445 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.631417 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="597150a2-dcbe-4a84-9a68-41eb187cbce4" containerName="payload-processing" Oct 30 01:44:30.631607 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.631470 6221 cpu_manager.go:414] "containerMap: removing container" podUID="cda4f86d-d23b-4da5-824e-cacea3b08ab7" containerName="istio-proxy" Oct 30 01:44:30.631607 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.631478 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cda4f86d-d23b-4da5-824e-cacea3b08ab7" containerName="istio-proxy" Oct 30 01:44:30.631607 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.631490 6221 cpu_manager.go:414] "containerMap: removing container" podUID="cda4f86d-d23b-4da5-824e-cacea3b08ab7" containerName="istio-proxy" Oct 30 01:44:30.631607 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.631495 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cda4f86d-d23b-4da5-824e-cacea3b08ab7" containerName="istio-proxy" Oct 30 01:44:30.631607 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.631506 6221 cpu_manager.go:414] "containerMap: removing container" podUID="73b85ea4-7469-459c-8ccd-3e9c669bac1c" containerName="istio-proxy" Oct 30 01:44:30.631607 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.631512 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73b85ea4-7469-459c-8ccd-3e9c669bac1c" containerName="istio-proxy" Oct 30 01:44:30.631607 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.631519 6221 cpu_manager.go:414] "containerMap: removing container" podUID="8f2b1f15-868f-4126-bf67-b65a2e5b50a7" containerName="payload-pre-processing" Oct 30 01:44:30.631607 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.631524 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f2b1f15-868f-4126-bf67-b65a2e5b50a7" containerName="payload-pre-processing" Oct 30 01:44:30.632079 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.631624 6221 cpu_manager.go:414] "containerMap: removing container" podUID="f0f39f0a-f1f1-41aa-a9d9-4df2e419ad59" containerName="payload-pre-processing" Oct 30 01:44:30.632079 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.631631 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0f39f0a-f1f1-41aa-a9d9-4df2e419ad59" containerName="payload-pre-processing" Oct 30 01:44:30.632079 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.631723 6221 cpu_manager.go:414] "containerMap: removing container" podUID="aa2ee40b-7d02-4d12-9894-7d4f51053320" containerName="main" Oct 30 01:44:30.632079 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.631731 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa2ee40b-7d02-4d12-9894-7d4f51053320" containerName="main" Oct 30 01:44:30.632079 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.631741 6221 cpu_manager.go:414] "containerMap: removing container" podUID="cec66c15-dfbf-4cc8-b4e7-bc42bdc45bae" containerName="payload-pre-processing" Oct 30 01:44:30.632079 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.631747 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cec66c15-dfbf-4cc8-b4e7-bc42bdc45bae" containerName="payload-pre-processing" Oct 30 01:44:30.632079 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.631766 6221 cpu_manager.go:414] "containerMap: removing container" podUID="f2416576-64de-4ee6-8082-52b1c1bb5e91" containerName="maas-api" Oct 30 01:44:30.632079 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.631771 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2416576-64de-4ee6-8082-52b1c1bb5e91" containerName="maas-api" Oct 30 01:44:30.632079 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.631777 6221 cpu_manager.go:414] "containerMap: removing container" podUID="74c960b2-ff58-455b-8eb4-43cc4171e32c" containerName="payload-processing" Oct 30 01:44:30.632079 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.631782 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74c960b2-ff58-455b-8eb4-43cc4171e32c" containerName="payload-processing" Oct 30 01:44:30.632079 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.631901 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa2ee40b-7d02-4d12-9894-7d4f51053320" containerName="main" Oct 30 01:44:30.632079 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.631910 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="cec66c15-dfbf-4cc8-b4e7-bc42bdc45bae" containerName="payload-pre-processing" Oct 30 01:44:30.632079 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.631919 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2416576-64de-4ee6-8082-52b1c1bb5e91" containerName="maas-api" Oct 30 01:44:30.632079 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.631924 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="74c960b2-ff58-455b-8eb4-43cc4171e32c" containerName="payload-processing" Oct 30 01:44:30.632079 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.631930 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="597150a2-dcbe-4a84-9a68-41eb187cbce4" containerName="payload-processing" Oct 30 01:44:30.632079 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.631937 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="cda4f86d-d23b-4da5-824e-cacea3b08ab7" containerName="istio-proxy" Oct 30 01:44:30.632079 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.631943 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="cda4f86d-d23b-4da5-824e-cacea3b08ab7" containerName="istio-proxy" Oct 30 01:44:30.632079 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.631949 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="73b85ea4-7469-459c-8ccd-3e9c669bac1c" containerName="istio-proxy" Oct 30 01:44:30.632079 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.631956 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f2b1f15-868f-4126-bf67-b65a2e5b50a7" containerName="payload-pre-processing" Oct 30 01:44:30.632079 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.631963 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0f39f0a-f1f1-41aa-a9d9-4df2e419ad59" containerName="payload-pre-processing" Oct 30 01:44:30.633068 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.633026 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f01cbc-5594bbb79d-bhxg9" podUID="f2416576-64de-4ee6-8082-52b1c1bb5e91" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.109:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:44:30.637222 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.637203 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-0ae258/praxis-route-0ae258-kserve-fb65bffff-s8cc2" Oct 30 01:44:30.662142 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.662115 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-0ae258/praxis-route-0ae258-kserve-fb65bffff-s8cc2"] Oct 30 01:44:30.669342 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.669315 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-0ae258\"/\"openshift-service-ca.crt\"" Oct 30 01:44:30.669342 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.669333 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-0ae258\"/\"kube-root-ca.crt\"" Oct 30 01:44:30.707139 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.707114 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-0ae258\"/\"default-dockercfg-ll6v5\"" Oct 30 01:44:30.714661 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.714627 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-0ae258\"/\"praxis-route-0ae258-kserve-self-signed-certs\"" Oct 30 01:44:30.719092 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.719071 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/532d4bd3-5ece-42da-876f-496110860d4d-model-cache\") pod \"praxis-route-0ae258-kserve-fb65bffff-s8cc2\" (UID: \"532d4bd3-5ece-42da-876f-496110860d4d\") " pod="ai-tenant-e2e-praxis-a-0ae258/praxis-route-0ae258-kserve-fb65bffff-s8cc2" Oct 30 01:44:30.719150 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.719116 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/532d4bd3-5ece-42da-876f-496110860d4d-dshm\") pod \"praxis-route-0ae258-kserve-fb65bffff-s8cc2\" (UID: \"532d4bd3-5ece-42da-876f-496110860d4d\") " pod="ai-tenant-e2e-praxis-a-0ae258/praxis-route-0ae258-kserve-fb65bffff-s8cc2" Oct 30 01:44:30.719263 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.719246 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/532d4bd3-5ece-42da-876f-496110860d4d-tls-certs\") pod \"praxis-route-0ae258-kserve-fb65bffff-s8cc2\" (UID: \"532d4bd3-5ece-42da-876f-496110860d4d\") " pod="ai-tenant-e2e-praxis-a-0ae258/praxis-route-0ae258-kserve-fb65bffff-s8cc2" Oct 30 01:44:30.719305 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.719287 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg96r\" (UniqueName: \"kubernetes.io/projected/532d4bd3-5ece-42da-876f-496110860d4d-kube-api-access-hg96r\") pod \"praxis-route-0ae258-kserve-fb65bffff-s8cc2\" (UID: \"532d4bd3-5ece-42da-876f-496110860d4d\") " pod="ai-tenant-e2e-praxis-a-0ae258/praxis-route-0ae258-kserve-fb65bffff-s8cc2" Oct 30 01:44:30.719343 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.719307 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/532d4bd3-5ece-42da-876f-496110860d4d-home\") pod \"praxis-route-0ae258-kserve-fb65bffff-s8cc2\" (UID: \"532d4bd3-5ece-42da-876f-496110860d4d\") " pod="ai-tenant-e2e-praxis-a-0ae258/praxis-route-0ae258-kserve-fb65bffff-s8cc2" Oct 30 01:44:30.719502 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.719485 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/532d4bd3-5ece-42da-876f-496110860d4d-tmp-dir\") pod \"praxis-route-0ae258-kserve-fb65bffff-s8cc2\" (UID: \"532d4bd3-5ece-42da-876f-496110860d4d\") " pod="ai-tenant-e2e-praxis-a-0ae258/praxis-route-0ae258-kserve-fb65bffff-s8cc2" Oct 30 01:44:30.820910 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.820850 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/532d4bd3-5ece-42da-876f-496110860d4d-tmp-dir\") pod \"praxis-route-0ae258-kserve-fb65bffff-s8cc2\" (UID: \"532d4bd3-5ece-42da-876f-496110860d4d\") " pod="ai-tenant-e2e-praxis-a-0ae258/praxis-route-0ae258-kserve-fb65bffff-s8cc2" Oct 30 01:44:30.820910 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.820915 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/532d4bd3-5ece-42da-876f-496110860d4d-model-cache\") pod \"praxis-route-0ae258-kserve-fb65bffff-s8cc2\" (UID: \"532d4bd3-5ece-42da-876f-496110860d4d\") " pod="ai-tenant-e2e-praxis-a-0ae258/praxis-route-0ae258-kserve-fb65bffff-s8cc2" Oct 30 01:44:30.821169 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.820969 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/532d4bd3-5ece-42da-876f-496110860d4d-dshm\") pod \"praxis-route-0ae258-kserve-fb65bffff-s8cc2\" (UID: \"532d4bd3-5ece-42da-876f-496110860d4d\") " pod="ai-tenant-e2e-praxis-a-0ae258/praxis-route-0ae258-kserve-fb65bffff-s8cc2" Oct 30 01:44:30.821169 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.821031 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/532d4bd3-5ece-42da-876f-496110860d4d-tls-certs\") pod \"praxis-route-0ae258-kserve-fb65bffff-s8cc2\" (UID: \"532d4bd3-5ece-42da-876f-496110860d4d\") " pod="ai-tenant-e2e-praxis-a-0ae258/praxis-route-0ae258-kserve-fb65bffff-s8cc2" Oct 30 01:44:30.821169 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.821070 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg96r\" (UniqueName: \"kubernetes.io/projected/532d4bd3-5ece-42da-876f-496110860d4d-kube-api-access-hg96r\") pod \"praxis-route-0ae258-kserve-fb65bffff-s8cc2\" (UID: \"532d4bd3-5ece-42da-876f-496110860d4d\") " pod="ai-tenant-e2e-praxis-a-0ae258/praxis-route-0ae258-kserve-fb65bffff-s8cc2" Oct 30 01:44:30.821169 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.821100 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/532d4bd3-5ece-42da-876f-496110860d4d-home\") pod \"praxis-route-0ae258-kserve-fb65bffff-s8cc2\" (UID: \"532d4bd3-5ece-42da-876f-496110860d4d\") " pod="ai-tenant-e2e-praxis-a-0ae258/praxis-route-0ae258-kserve-fb65bffff-s8cc2" Oct 30 01:44:30.821378 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.821262 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/532d4bd3-5ece-42da-876f-496110860d4d-tmp-dir\") pod \"praxis-route-0ae258-kserve-fb65bffff-s8cc2\" (UID: \"532d4bd3-5ece-42da-876f-496110860d4d\") " pod="ai-tenant-e2e-praxis-a-0ae258/praxis-route-0ae258-kserve-fb65bffff-s8cc2" Oct 30 01:44:30.821378 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.821319 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/532d4bd3-5ece-42da-876f-496110860d4d-model-cache\") pod \"praxis-route-0ae258-kserve-fb65bffff-s8cc2\" (UID: \"532d4bd3-5ece-42da-876f-496110860d4d\") " pod="ai-tenant-e2e-praxis-a-0ae258/praxis-route-0ae258-kserve-fb65bffff-s8cc2" Oct 30 01:44:30.821605 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.821577 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/532d4bd3-5ece-42da-876f-496110860d4d-home\") pod \"praxis-route-0ae258-kserve-fb65bffff-s8cc2\" (UID: \"532d4bd3-5ece-42da-876f-496110860d4d\") " pod="ai-tenant-e2e-praxis-a-0ae258/praxis-route-0ae258-kserve-fb65bffff-s8cc2" Oct 30 01:44:30.823718 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.823685 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/532d4bd3-5ece-42da-876f-496110860d4d-dshm\") pod \"praxis-route-0ae258-kserve-fb65bffff-s8cc2\" (UID: \"532d4bd3-5ece-42da-876f-496110860d4d\") " pod="ai-tenant-e2e-praxis-a-0ae258/praxis-route-0ae258-kserve-fb65bffff-s8cc2" Oct 30 01:44:30.823854 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.823838 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/532d4bd3-5ece-42da-876f-496110860d4d-tls-certs\") pod \"praxis-route-0ae258-kserve-fb65bffff-s8cc2\" (UID: \"532d4bd3-5ece-42da-876f-496110860d4d\") " pod="ai-tenant-e2e-praxis-a-0ae258/praxis-route-0ae258-kserve-fb65bffff-s8cc2" Oct 30 01:44:30.844712 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.844639 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg96r\" (UniqueName: \"kubernetes.io/projected/532d4bd3-5ece-42da-876f-496110860d4d-kube-api-access-hg96r\") pod \"praxis-route-0ae258-kserve-fb65bffff-s8cc2\" (UID: \"532d4bd3-5ece-42da-876f-496110860d4d\") " pod="ai-tenant-e2e-praxis-a-0ae258/praxis-route-0ae258-kserve-fb65bffff-s8cc2" Oct 30 01:44:30.960238 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:30.959964 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-0ae258/praxis-route-0ae258-kserve-fb65bffff-s8cc2" Oct 30 01:44:31.355123 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:31.355033 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-0ae258/praxis-route-0ae258-kserve-fb65bffff-s8cc2"] Oct 30 01:44:31.358243 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:44:31.358215 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod532d4bd3_5ece_42da_876f_496110860d4d.slice/crio-6496eccc167b45e6cea1697ced6e2a8a56df41100cd2875a3135afa1abed4bfd WatchSource:0}: Error finding container 6496eccc167b45e6cea1697ced6e2a8a56df41100cd2875a3135afa1abed4bfd: Status 404 returned error can't find the container with id 6496eccc167b45e6cea1697ced6e2a8a56df41100cd2875a3135afa1abed4bfd Oct 30 01:44:31.436648 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:31.436617 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-0ae258/praxis-route-0ae258-kserve-fb65bffff-s8cc2" event={"ID":"532d4bd3-5ece-42da-876f-496110860d4d","Type":"ContainerStarted","Data":"26e018a2b9923dd6ecfab292bd09228395712887bec5204add579f413598c96c"} Oct 30 01:44:31.436796 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:31.436657 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-0ae258/praxis-route-0ae258-kserve-fb65bffff-s8cc2" event={"ID":"532d4bd3-5ece-42da-876f-496110860d4d","Type":"ContainerStarted","Data":"6496eccc167b45e6cea1697ced6e2a8a56df41100cd2875a3135afa1abed4bfd"} Oct 30 01:44:31.478203 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:31.478145 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-0ae258/praxis-route-0ae258-kserve-fb65bffff-s8cc2" podStartSLOduration=1.478132173 podStartE2EDuration="1.478132173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:44:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:44:31.477901693 +0000 UTC m=+1595.056189030" watchObservedRunningTime="2026-10-30 01:44:31.478132173 +0000 UTC m=+1595.056419509" Oct 30 01:44:31.961115 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:31.961084 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-0ae258/praxis-route-0ae258-kserve-fb65bffff-s8cc2" Oct 30 01:44:32.314323 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:32.314292 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-be8a7d-67797b4964-srsbd"] Oct 30 01:44:32.378063 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:32.378028 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-be8a7d-5b68d984ff-5bh77"] Oct 30 01:44:34.391672 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:34.391637 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-be8a7d-5dcd7876bd-7h9q5"] Oct 30 01:44:34.392105 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:34.391872 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-be8a7d-5dcd7876bd-7h9q5" podUID="b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b" containerName="maas-api" containerID="cri-o://d2da69f4b3787a0263963a6dc747e2aa3e8bf8547389394a5b9df25654bf7507" gracePeriod=30 Oct 30 01:44:34.762860 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:34.762832 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-be8a7d-5dcd7876bd-7h9q5" Oct 30 01:44:34.888279 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:34.886967 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b-openshift-service-ca\") pod \"b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b\" (UID: \"b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b\") " Oct 30 01:44:34.888279 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:34.887033 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b-kube-api-access-nw6fj\" (UniqueName: \"kubernetes.io/projected/b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b-kube-api-access-nw6fj\") pod \"b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b\" (UID: \"b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b\") " Oct 30 01:44:34.888279 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:34.887097 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b-maas-api-tls\") pod \"b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b\" (UID: \"b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b\") " Oct 30 01:44:34.888279 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:34.887283 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b-metrics-certs\") pod \"b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b\" (UID: \"b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b\") " Oct 30 01:44:34.888640 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:34.888499 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b-openshift-service-ca" pod "b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b" (UID: "b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:44:34.907871 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:34.896734 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b-metrics-certs" pod "b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b" (UID: "b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:44:34.911156 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:34.911112 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b-kube-api-access-nw6fj" pod "b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b" (UID: "b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b"). InnerVolumeSpecName "kube-api-access-nw6fj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:44:34.921032 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:34.920991 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b-maas-api-tls" pod "b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b" (UID: "b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:44:34.989206 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:34.989132 6221 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b-openshift-service-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:44:34.989206 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:34.989160 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw6fj\" (UniqueName: \"kubernetes.io/projected/b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b-kube-api-access-nw6fj\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:44:34.989206 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:34.989172 6221 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b-maas-api-tls\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:44:34.989206 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:34.989182 6221 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b-metrics-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:44:35.468979 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:35.468942 6221 generic.go:356] "Generic (PLEG): container finished" podID="b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b" containerID="d2da69f4b3787a0263963a6dc747e2aa3e8bf8547389394a5b9df25654bf7507" exitCode=0 Oct 30 01:44:35.469453 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:35.468987 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-be8a7d-5dcd7876bd-7h9q5" Oct 30 01:44:35.469453 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:35.468993 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-be8a7d-5dcd7876bd-7h9q5" event={"ID":"b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b","Type":"ContainerDied","Data":"d2da69f4b3787a0263963a6dc747e2aa3e8bf8547389394a5b9df25654bf7507"} Oct 30 01:44:35.469453 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:35.469043 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-be8a7d-5dcd7876bd-7h9q5" event={"ID":"b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b","Type":"ContainerDied","Data":"6f6aad89d86ac035e323e7e18a210eaac38fe4f500e30a1632ab6e6eb2783fb3"} Oct 30 01:44:35.469453 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:35.469076 6221 scope.go:122] "RemoveContainer" containerID="d2da69f4b3787a0263963a6dc747e2aa3e8bf8547389394a5b9df25654bf7507" Oct 30 01:44:35.479115 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:35.479094 6221 scope.go:122] "RemoveContainer" containerID="d2da69f4b3787a0263963a6dc747e2aa3e8bf8547389394a5b9df25654bf7507" Oct 30 01:44:35.479361 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:44:35.479342 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2da69f4b3787a0263963a6dc747e2aa3e8bf8547389394a5b9df25654bf7507\": container with ID starting with d2da69f4b3787a0263963a6dc747e2aa3e8bf8547389394a5b9df25654bf7507 not found: ID does not exist" containerID="d2da69f4b3787a0263963a6dc747e2aa3e8bf8547389394a5b9df25654bf7507" Oct 30 01:44:35.479409 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:35.479368 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2da69f4b3787a0263963a6dc747e2aa3e8bf8547389394a5b9df25654bf7507"} err="failed to get container status \"d2da69f4b3787a0263963a6dc747e2aa3e8bf8547389394a5b9df25654bf7507\": rpc error: code = NotFound desc = could not find container \"d2da69f4b3787a0263963a6dc747e2aa3e8bf8547389394a5b9df25654bf7507\": container with ID starting with d2da69f4b3787a0263963a6dc747e2aa3e8bf8547389394a5b9df25654bf7507 not found: ID does not exist" Oct 30 01:44:35.557308 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:35.557267 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-be8a7d-5dcd7876bd-7h9q5"] Oct 30 01:44:35.584553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:35.581077 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-be8a7d-5dcd7876bd-7h9q5"] Oct 30 01:44:36.547250 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:36.547203 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b" path="/var/lib/kubelet/pods/b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b/volumes" Oct 30 01:44:37.314641 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:37.314581 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-be8a7d-67797b4964-srsbd" podUID="b3429734-086d-499f-b92c-28c980fdbd66" containerName="payload-processing" containerID="cri-o://3fa9f453f1473361626ee9004d2318f0440f9c779b4a534fd98364d93f27a32b" gracePeriod=25 Oct 30 01:44:37.379084 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:37.379046 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-be8a7d-5b68d984ff-5bh77" podUID="7ec92517-e71d-4a1c-a142-000fd54b090c" containerName="payload-pre-processing" containerID="cri-o://7756bf99bbe93f8131811aeb7e29a32f76dd90b91ca02629123302dfcb3512ef" gracePeriod=25 Oct 30 01:44:37.519677 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:37.519647 6221 generic.go:356] "Generic (PLEG): container finished" podID="b3429734-086d-499f-b92c-28c980fdbd66" containerID="3fa9f453f1473361626ee9004d2318f0440f9c779b4a534fd98364d93f27a32b" exitCode=0 Oct 30 01:44:37.519788 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:37.519727 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-be8a7d-67797b4964-srsbd" event={"ID":"b3429734-086d-499f-b92c-28c980fdbd66","Type":"ContainerDied","Data":"3fa9f453f1473361626ee9004d2318f0440f9c779b4a534fd98364d93f27a32b"} Oct 30 01:44:37.688220 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:37.688193 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-be8a7d-67797b4964-srsbd" Oct 30 01:44:37.702000 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:37.701975 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-be8a7d-5b68d984ff-5bh77" Oct 30 01:44:37.845904 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:37.845814 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3429734-086d-499f-b92c-28c980fdbd66-kube-api-access-phnw9\" (UniqueName: \"kubernetes.io/projected/b3429734-086d-499f-b92c-28c980fdbd66-kube-api-access-phnw9\") pod \"b3429734-086d-499f-b92c-28c980fdbd66\" (UID: \"b3429734-086d-499f-b92c-28c980fdbd66\") " Oct 30 01:44:37.846070 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:37.845963 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3429734-086d-499f-b92c-28c980fdbd66-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3429734-086d-499f-b92c-28c980fdbd66-plugins-config-volume\") pod \"b3429734-086d-499f-b92c-28c980fdbd66\" (UID: \"b3429734-086d-499f-b92c-28c980fdbd66\") " Oct 30 01:44:37.846070 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:37.845999 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ec92517-e71d-4a1c-a142-000fd54b090c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ec92517-e71d-4a1c-a142-000fd54b090c-plugins-config-volume\") pod \"7ec92517-e71d-4a1c-a142-000fd54b090c\" (UID: \"7ec92517-e71d-4a1c-a142-000fd54b090c\") " Oct 30 01:44:37.846210 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:37.846097 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ec92517-e71d-4a1c-a142-000fd54b090c-kube-api-access-mx56s\" (UniqueName: \"kubernetes.io/projected/7ec92517-e71d-4a1c-a142-000fd54b090c-kube-api-access-mx56s\") pod \"7ec92517-e71d-4a1c-a142-000fd54b090c\" (UID: \"7ec92517-e71d-4a1c-a142-000fd54b090c\") " Oct 30 01:44:37.846412 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:37.846389 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3429734-086d-499f-b92c-28c980fdbd66-plugins-config-volume" pod "b3429734-086d-499f-b92c-28c980fdbd66" (UID: "b3429734-086d-499f-b92c-28c980fdbd66"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:44:37.846469 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:37.846423 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ec92517-e71d-4a1c-a142-000fd54b090c-plugins-config-volume" pod "7ec92517-e71d-4a1c-a142-000fd54b090c" (UID: "7ec92517-e71d-4a1c-a142-000fd54b090c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:44:37.846728 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:37.846708 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3429734-086d-499f-b92c-28c980fdbd66-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:44:37.846789 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:37.846736 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ec92517-e71d-4a1c-a142-000fd54b090c-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:44:37.848153 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:37.848129 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ec92517-e71d-4a1c-a142-000fd54b090c-kube-api-access-mx56s" pod "7ec92517-e71d-4a1c-a142-000fd54b090c" (UID: "7ec92517-e71d-4a1c-a142-000fd54b090c"). InnerVolumeSpecName "kube-api-access-mx56s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:44:37.848153 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:37.848142 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3429734-086d-499f-b92c-28c980fdbd66-kube-api-access-phnw9" pod "b3429734-086d-499f-b92c-28c980fdbd66" (UID: "b3429734-086d-499f-b92c-28c980fdbd66"). InnerVolumeSpecName "kube-api-access-phnw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:44:37.947790 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:37.947752 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx56s\" (UniqueName: \"kubernetes.io/projected/7ec92517-e71d-4a1c-a142-000fd54b090c-kube-api-access-mx56s\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:44:37.947790 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:37.947784 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phnw9\" (UniqueName: \"kubernetes.io/projected/b3429734-086d-499f-b92c-28c980fdbd66-kube-api-access-phnw9\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:44:38.569242 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:38.569198 6221 generic.go:356] "Generic (PLEG): container finished" podID="7ec92517-e71d-4a1c-a142-000fd54b090c" containerID="7756bf99bbe93f8131811aeb7e29a32f76dd90b91ca02629123302dfcb3512ef" exitCode=0 Oct 30 01:44:38.569457 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:38.569351 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-be8a7d-5b68d984ff-5bh77" Oct 30 01:44:38.576332 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:38.576013 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-be8a7d-67797b4964-srsbd" Oct 30 01:44:38.578125 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:38.578094 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-be8a7d-5b68d984ff-5bh77" event={"ID":"7ec92517-e71d-4a1c-a142-000fd54b090c","Type":"ContainerDied","Data":"7756bf99bbe93f8131811aeb7e29a32f76dd90b91ca02629123302dfcb3512ef"} Oct 30 01:44:38.578254 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:38.578149 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-be8a7d-5b68d984ff-5bh77" event={"ID":"7ec92517-e71d-4a1c-a142-000fd54b090c","Type":"ContainerDied","Data":"a3e41a47594c3bcdf3b53bfe2db32819b6ed2ab6575c68ade82d197999b40cae"} Oct 30 01:44:38.578254 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:38.578171 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-be8a7d-67797b4964-srsbd" event={"ID":"b3429734-086d-499f-b92c-28c980fdbd66","Type":"ContainerDied","Data":"333f1955b084b9c7337c041a5592a4167a2773b6897ca4f2fbd7e65a3d0905a4"} Oct 30 01:44:38.578254 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:38.578201 6221 scope.go:122] "RemoveContainer" containerID="7756bf99bbe93f8131811aeb7e29a32f76dd90b91ca02629123302dfcb3512ef" Oct 30 01:44:38.595999 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:38.595974 6221 scope.go:122] "RemoveContainer" containerID="7756bf99bbe93f8131811aeb7e29a32f76dd90b91ca02629123302dfcb3512ef" Oct 30 01:44:38.596283 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:44:38.596262 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7756bf99bbe93f8131811aeb7e29a32f76dd90b91ca02629123302dfcb3512ef\": container with ID starting with 7756bf99bbe93f8131811aeb7e29a32f76dd90b91ca02629123302dfcb3512ef not found: ID does not exist" containerID="7756bf99bbe93f8131811aeb7e29a32f76dd90b91ca02629123302dfcb3512ef" Oct 30 01:44:38.596330 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:38.596292 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7756bf99bbe93f8131811aeb7e29a32f76dd90b91ca02629123302dfcb3512ef"} err="failed to get container status \"7756bf99bbe93f8131811aeb7e29a32f76dd90b91ca02629123302dfcb3512ef\": rpc error: code = NotFound desc = could not find container \"7756bf99bbe93f8131811aeb7e29a32f76dd90b91ca02629123302dfcb3512ef\": container with ID starting with 7756bf99bbe93f8131811aeb7e29a32f76dd90b91ca02629123302dfcb3512ef not found: ID does not exist" Oct 30 01:44:38.596330 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:38.596309 6221 scope.go:122] "RemoveContainer" containerID="3fa9f453f1473361626ee9004d2318f0440f9c779b4a534fd98364d93f27a32b" Oct 30 01:44:38.651135 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:38.651105 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-be8a7d-5b68d984ff-5bh77"] Oct 30 01:44:38.663895 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:38.663869 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-be8a7d-5b68d984ff-5bh77"] Oct 30 01:44:38.697206 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:38.697174 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-be8a7d-67797b4964-srsbd"] Oct 30 01:44:38.700611 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:38.700590 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-be8a7d-67797b4964-srsbd"] Oct 30 01:44:40.547098 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:40.547068 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ec92517-e71d-4a1c-a142-000fd54b090c" path="/var/lib/kubelet/pods/7ec92517-e71d-4a1c-a142-000fd54b090c/volumes" Oct 30 01:44:40.547481 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:40.547464 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3429734-086d-499f-b92c-28c980fdbd66" path="/var/lib/kubelet/pods/b3429734-086d-499f-b92c-28c980fdbd66/volumes" Oct 30 01:44:40.961306 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:40.961216 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-0ae258/praxis-route-0ae258-kserve-fb65bffff-s8cc2" Oct 30 01:44:40.973766 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:40.973739 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-0ae258/praxis-route-0ae258-kserve-fb65bffff-s8cc2" Oct 30 01:44:41.699739 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:41.699705 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-0ae258/praxis-route-0ae258-kserve-fb65bffff-s8cc2" Oct 30 01:44:47.720200 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:47.720163 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f197b1-794699547-t9qsc"] Oct 30 01:44:47.720742 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:47.720726 6221 cpu_manager.go:414] "containerMap: removing container" podUID="7ec92517-e71d-4a1c-a142-000fd54b090c" containerName="payload-pre-processing" Oct 30 01:44:47.720787 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:47.720743 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ec92517-e71d-4a1c-a142-000fd54b090c" containerName="payload-pre-processing" Oct 30 01:44:47.720787 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:47.720763 6221 cpu_manager.go:414] "containerMap: removing container" podUID="b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b" containerName="maas-api" Oct 30 01:44:47.720787 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:47.720772 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b" containerName="maas-api" Oct 30 01:44:47.720894 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:47.720795 6221 cpu_manager.go:414] "containerMap: removing container" podUID="b3429734-086d-499f-b92c-28c980fdbd66" containerName="payload-processing" Oct 30 01:44:47.720894 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:47.720800 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3429734-086d-499f-b92c-28c980fdbd66" containerName="payload-processing" Oct 30 01:44:47.720894 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:47.720881 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3429734-086d-499f-b92c-28c980fdbd66" containerName="payload-processing" Oct 30 01:44:47.720894 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:47.720895 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ec92517-e71d-4a1c-a142-000fd54b090c" containerName="payload-pre-processing" Oct 30 01:44:47.721012 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:47.720902 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8d53843-bfe3-41a4-92c9-3c7f2ba9eb1b" containerName="maas-api" Oct 30 01:44:47.724243 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:47.724226 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f197b1-794699547-t9qsc" Oct 30 01:44:47.844552 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:47.841892 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-f197b1-dockercfg-h2wtb\"" Oct 30 01:44:47.844552 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:47.842002 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-f197b1\"" Oct 30 01:44:47.851570 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:47.845674 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a217cae-637d-4f32-908b-b9ea322a01e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-f197b1-794699547-t9qsc\" (UID: \"4a217cae-637d-4f32-908b-b9ea322a01e7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f197b1-794699547-t9qsc" Oct 30 01:44:47.860553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:47.851901 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkxtv\" (UniqueName: \"kubernetes.io/projected/4a217cae-637d-4f32-908b-b9ea322a01e7-kube-api-access-qkxtv\") pod \"payload-pre-processing-e2e-praxis-a-f197b1-794699547-t9qsc\" (UID: \"4a217cae-637d-4f32-908b-b9ea322a01e7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f197b1-794699547-t9qsc" Oct 30 01:44:47.892009 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:47.891950 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f197b1-794699547-t9qsc"] Oct 30 01:44:47.961979 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:47.961939 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a217cae-637d-4f32-908b-b9ea322a01e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-f197b1-794699547-t9qsc\" (UID: \"4a217cae-637d-4f32-908b-b9ea322a01e7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f197b1-794699547-t9qsc" Oct 30 01:44:47.962158 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:47.962021 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkxtv\" (UniqueName: \"kubernetes.io/projected/4a217cae-637d-4f32-908b-b9ea322a01e7-kube-api-access-qkxtv\") pod \"payload-pre-processing-e2e-praxis-a-f197b1-794699547-t9qsc\" (UID: \"4a217cae-637d-4f32-908b-b9ea322a01e7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f197b1-794699547-t9qsc" Oct 30 01:44:47.962753 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:47.962734 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a217cae-637d-4f32-908b-b9ea322a01e7-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-f197b1-794699547-t9qsc\" (UID: \"4a217cae-637d-4f32-908b-b9ea322a01e7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f197b1-794699547-t9qsc" Oct 30 01:44:47.983885 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:47.983819 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkxtv\" (UniqueName: \"kubernetes.io/projected/4a217cae-637d-4f32-908b-b9ea322a01e7-kube-api-access-qkxtv\") pod \"payload-pre-processing-e2e-praxis-a-f197b1-794699547-t9qsc\" (UID: \"4a217cae-637d-4f32-908b-b9ea322a01e7\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f197b1-794699547-t9qsc" Oct 30 01:44:48.037783 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:48.037748 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f197b1-794699547-t9qsc" Oct 30 01:44:48.335482 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:48.335457 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f197b1-794699547-t9qsc"] Oct 30 01:44:48.338341 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:44:48.338309 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a217cae_637d_4f32_908b_b9ea322a01e7.slice/crio-4e616faf15975d7c190108f8787088b14c7205c756d86d0994eb8332cb867894 WatchSource:0}: Error finding container 4e616faf15975d7c190108f8787088b14c7205c756d86d0994eb8332cb867894: Status 404 returned error can't find the container with id 4e616faf15975d7c190108f8787088b14c7205c756d86d0994eb8332cb867894 Oct 30 01:44:48.648679 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:48.648589 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f197b1-794699547-t9qsc" event={"ID":"4a217cae-637d-4f32-908b-b9ea322a01e7","Type":"ContainerStarted","Data":"fce6adce05f07aa7776798e0b82b7536e898d97c6b04ef601f6cc34515a11531"} Oct 30 01:44:48.648679 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:48.648663 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f197b1-794699547-t9qsc" Oct 30 01:44:48.648858 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:48.648683 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f197b1-794699547-t9qsc" event={"ID":"4a217cae-637d-4f32-908b-b9ea322a01e7","Type":"ContainerStarted","Data":"4e616faf15975d7c190108f8787088b14c7205c756d86d0994eb8332cb867894"} Oct 30 01:44:48.687808 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:48.687748 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f197b1-794699547-t9qsc" podStartSLOduration=1.687734557 podStartE2EDuration="1.687734557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:44:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:44:48.686987377 +0000 UTC m=+1612.265274732" watchObservedRunningTime="2026-10-30 01:44:48.687734557 +0000 UTC m=+1612.266021894" Oct 30 01:44:51.494374 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:44:51.494336 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:44:52.017103 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:52.017071 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:44:59.658436 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:44:59.658403 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f197b1-794699547-t9qsc" Oct 30 01:45:00.312975 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:00.312939 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f197b1-5847f96fcd-4frnn"] Oct 30 01:45:00.318844 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:00.318818 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f197b1-5847f96fcd-4frnn" Oct 30 01:45:00.336453 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:00.336425 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f197b1-5847f96fcd-4frnn"] Oct 30 01:45:00.345390 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:00.345359 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-f197b1-serving-cert\"" Oct 30 01:45:00.359296 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:00.359249 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d44c10f7-7694-4fdd-86ce-669947932edb-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-f197b1-5847f96fcd-4frnn\" (UID: \"d44c10f7-7694-4fdd-86ce-669947932edb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f197b1-5847f96fcd-4frnn" Oct 30 01:45:00.359466 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:00.359315 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2p6c\" (UniqueName: \"kubernetes.io/projected/d44c10f7-7694-4fdd-86ce-669947932edb-kube-api-access-z2p6c\") pod \"maas-api-e2e-praxis-a-f197b1-5847f96fcd-4frnn\" (UID: \"d44c10f7-7694-4fdd-86ce-669947932edb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f197b1-5847f96fcd-4frnn" Oct 30 01:45:00.359466 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:00.359438 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d44c10f7-7694-4fdd-86ce-669947932edb-maas-api-tls\") pod \"maas-api-e2e-praxis-a-f197b1-5847f96fcd-4frnn\" (UID: \"d44c10f7-7694-4fdd-86ce-669947932edb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f197b1-5847f96fcd-4frnn" Oct 30 01:45:00.359574 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:00.359503 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d44c10f7-7694-4fdd-86ce-669947932edb-metrics-certs\") pod \"maas-api-e2e-praxis-a-f197b1-5847f96fcd-4frnn\" (UID: \"d44c10f7-7694-4fdd-86ce-669947932edb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f197b1-5847f96fcd-4frnn" Oct 30 01:45:00.461350 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:00.461309 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d44c10f7-7694-4fdd-86ce-669947932edb-metrics-certs\") pod \"maas-api-e2e-praxis-a-f197b1-5847f96fcd-4frnn\" (UID: \"d44c10f7-7694-4fdd-86ce-669947932edb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f197b1-5847f96fcd-4frnn" Oct 30 01:45:00.461607 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:00.461501 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d44c10f7-7694-4fdd-86ce-669947932edb-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-f197b1-5847f96fcd-4frnn\" (UID: \"d44c10f7-7694-4fdd-86ce-669947932edb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f197b1-5847f96fcd-4frnn" Oct 30 01:45:00.461607 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:00.461589 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2p6c\" (UniqueName: \"kubernetes.io/projected/d44c10f7-7694-4fdd-86ce-669947932edb-kube-api-access-z2p6c\") pod \"maas-api-e2e-praxis-a-f197b1-5847f96fcd-4frnn\" (UID: \"d44c10f7-7694-4fdd-86ce-669947932edb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f197b1-5847f96fcd-4frnn" Oct 30 01:45:00.461756 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:00.461644 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d44c10f7-7694-4fdd-86ce-669947932edb-maas-api-tls\") pod \"maas-api-e2e-praxis-a-f197b1-5847f96fcd-4frnn\" (UID: \"d44c10f7-7694-4fdd-86ce-669947932edb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f197b1-5847f96fcd-4frnn" Oct 30 01:45:00.461938 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:45:00.461914 6221 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-f197b1-serving-cert: secret "maas-api-e2e-praxis-a-f197b1-serving-cert" not found Oct 30 01:45:00.462044 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:45:00.462007 6221 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d44c10f7-7694-4fdd-86ce-669947932edb-maas-api-tls podName:d44c10f7-7694-4fdd-86ce-669947932edb nodeName:}" failed. No retries permitted until 2026-10-30 01:45:00.961987811 +0000 UTC m=+1624.540275135 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d44c10f7-7694-4fdd-86ce-669947932edb-maas-api-tls") pod "maas-api-e2e-praxis-a-f197b1-5847f96fcd-4frnn" (UID: "d44c10f7-7694-4fdd-86ce-669947932edb") : secret "maas-api-e2e-praxis-a-f197b1-serving-cert" not found Oct 30 01:45:00.462227 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:00.462199 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d44c10f7-7694-4fdd-86ce-669947932edb-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-f197b1-5847f96fcd-4frnn\" (UID: \"d44c10f7-7694-4fdd-86ce-669947932edb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f197b1-5847f96fcd-4frnn" Oct 30 01:45:00.463908 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:00.463890 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d44c10f7-7694-4fdd-86ce-669947932edb-metrics-certs\") pod \"maas-api-e2e-praxis-a-f197b1-5847f96fcd-4frnn\" (UID: \"d44c10f7-7694-4fdd-86ce-669947932edb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f197b1-5847f96fcd-4frnn" Oct 30 01:45:00.487517 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:00.487489 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2p6c\" (UniqueName: \"kubernetes.io/projected/d44c10f7-7694-4fdd-86ce-669947932edb-kube-api-access-z2p6c\") pod \"maas-api-e2e-praxis-a-f197b1-5847f96fcd-4frnn\" (UID: \"d44c10f7-7694-4fdd-86ce-669947932edb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f197b1-5847f96fcd-4frnn" Oct 30 01:45:00.972833 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:00.972796 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d44c10f7-7694-4fdd-86ce-669947932edb-maas-api-tls\") pod \"maas-api-e2e-praxis-a-f197b1-5847f96fcd-4frnn\" (UID: \"d44c10f7-7694-4fdd-86ce-669947932edb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f197b1-5847f96fcd-4frnn" Oct 30 01:45:00.975232 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:00.975207 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d44c10f7-7694-4fdd-86ce-669947932edb-maas-api-tls\") pod \"maas-api-e2e-praxis-a-f197b1-5847f96fcd-4frnn\" (UID: \"d44c10f7-7694-4fdd-86ce-669947932edb\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f197b1-5847f96fcd-4frnn" Oct 30 01:45:01.232111 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:01.232031 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f197b1-5847f96fcd-4frnn" Oct 30 01:45:01.577137 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:01.577107 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f197b1-5847f96fcd-4frnn"] Oct 30 01:45:01.579036 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:45:01.578986 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd44c10f7_7694_4fdd_86ce_669947932edb.slice/crio-caa87dedf2264fd759d0a850a1825b0c9ac1cf4996a668e96628913769aca69a WatchSource:0}: Error finding container caa87dedf2264fd759d0a850a1825b0c9ac1cf4996a668e96628913769aca69a: Status 404 returned error can't find the container with id caa87dedf2264fd759d0a850a1825b0c9ac1cf4996a668e96628913769aca69a Oct 30 01:45:01.745476 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:01.745435 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f197b1-5847f96fcd-4frnn" event={"ID":"d44c10f7-7694-4fdd-86ce-669947932edb","Type":"ContainerStarted","Data":"caa87dedf2264fd759d0a850a1825b0c9ac1cf4996a668e96628913769aca69a"} Oct 30 01:45:02.763119 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:02.763073 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f197b1-5847f96fcd-4frnn" event={"ID":"d44c10f7-7694-4fdd-86ce-669947932edb","Type":"ContainerStarted","Data":"14fc89219800f091e02610ce4ba6f579d858d093e4bf8f9364b8f42f7f5caebe"} Oct 30 01:45:02.763659 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:02.763257 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f197b1-5847f96fcd-4frnn" Oct 30 01:45:02.816501 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:02.816444 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f197b1-5847f96fcd-4frnn" podStartSLOduration=2.310450251 podStartE2EDuration="2.816429575s" totalImagesPullingTime="505.979324ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:02.814151293 +0000 UTC m=+1626.392438629" watchObservedRunningTime="2026-10-30 01:45:02.816429575 +0000 UTC m=+1626.394716911" Oct 30 01:45:08.775819 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:08.775743 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f197b1-5847f96fcd-4frnn" Oct 30 01:45:19.182731 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:19.182694 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f"] Oct 30 01:45:19.189282 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:19.186839 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f" Oct 30 01:45:19.220348 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:19.220316 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f"] Oct 30 01:45:19.274381 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:19.274346 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f347ecd6-e2d7-43da-ba3e-88006f24c167-istio-envoy\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f\" (UID: \"f347ecd6-e2d7-43da-ba3e-88006f24c167\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f" Oct 30 01:45:19.274591 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:19.274453 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmprs\" (UniqueName: \"kubernetes.io/projected/f347ecd6-e2d7-43da-ba3e-88006f24c167-kube-api-access-jmprs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f\" (UID: \"f347ecd6-e2d7-43da-ba3e-88006f24c167\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f" Oct 30 01:45:19.274682 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:19.274617 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f347ecd6-e2d7-43da-ba3e-88006f24c167-workload-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f\" (UID: \"f347ecd6-e2d7-43da-ba3e-88006f24c167\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f" Oct 30 01:45:19.274746 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:19.274726 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f347ecd6-e2d7-43da-ba3e-88006f24c167-workload-certs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f\" (UID: \"f347ecd6-e2d7-43da-ba3e-88006f24c167\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f" Oct 30 01:45:19.274801 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:19.274790 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f347ecd6-e2d7-43da-ba3e-88006f24c167-istio-token\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f\" (UID: \"f347ecd6-e2d7-43da-ba3e-88006f24c167\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f" Oct 30 01:45:19.274890 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:19.274873 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f347ecd6-e2d7-43da-ba3e-88006f24c167-istio-podinfo\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f\" (UID: \"f347ecd6-e2d7-43da-ba3e-88006f24c167\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f" Oct 30 01:45:19.274948 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:19.274936 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f347ecd6-e2d7-43da-ba3e-88006f24c167-istiod-ca-cert\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f\" (UID: \"f347ecd6-e2d7-43da-ba3e-88006f24c167\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f" Oct 30 01:45:19.274997 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:19.274972 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f347ecd6-e2d7-43da-ba3e-88006f24c167-credential-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f\" (UID: \"f347ecd6-e2d7-43da-ba3e-88006f24c167\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f" Oct 30 01:45:19.275055 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:19.275038 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f347ecd6-e2d7-43da-ba3e-88006f24c167-istio-data\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f\" (UID: \"f347ecd6-e2d7-43da-ba3e-88006f24c167\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f" Oct 30 01:45:19.376477 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:19.376440 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f347ecd6-e2d7-43da-ba3e-88006f24c167-istiod-ca-cert\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f\" (UID: \"f347ecd6-e2d7-43da-ba3e-88006f24c167\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f" Oct 30 01:45:19.376682 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:19.376487 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f347ecd6-e2d7-43da-ba3e-88006f24c167-credential-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f\" (UID: \"f347ecd6-e2d7-43da-ba3e-88006f24c167\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f" Oct 30 01:45:19.376682 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:19.376517 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f347ecd6-e2d7-43da-ba3e-88006f24c167-istio-data\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f\" (UID: \"f347ecd6-e2d7-43da-ba3e-88006f24c167\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f" Oct 30 01:45:19.376682 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:19.376611 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f347ecd6-e2d7-43da-ba3e-88006f24c167-istio-envoy\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f\" (UID: \"f347ecd6-e2d7-43da-ba3e-88006f24c167\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f" Oct 30 01:45:19.376682 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:19.376669 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmprs\" (UniqueName: \"kubernetes.io/projected/f347ecd6-e2d7-43da-ba3e-88006f24c167-kube-api-access-jmprs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f\" (UID: \"f347ecd6-e2d7-43da-ba3e-88006f24c167\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f" Oct 30 01:45:19.376918 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:19.376811 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f347ecd6-e2d7-43da-ba3e-88006f24c167-workload-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f\" (UID: \"f347ecd6-e2d7-43da-ba3e-88006f24c167\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f" Oct 30 01:45:19.376918 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:19.376901 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f347ecd6-e2d7-43da-ba3e-88006f24c167-workload-certs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f\" (UID: \"f347ecd6-e2d7-43da-ba3e-88006f24c167\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f" Oct 30 01:45:19.377027 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:19.376941 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f347ecd6-e2d7-43da-ba3e-88006f24c167-istio-token\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f\" (UID: \"f347ecd6-e2d7-43da-ba3e-88006f24c167\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f" Oct 30 01:45:19.377027 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:19.376992 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f347ecd6-e2d7-43da-ba3e-88006f24c167-istio-data\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f\" (UID: \"f347ecd6-e2d7-43da-ba3e-88006f24c167\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f" Oct 30 01:45:19.377130 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:19.377026 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f347ecd6-e2d7-43da-ba3e-88006f24c167-istio-podinfo\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f\" (UID: \"f347ecd6-e2d7-43da-ba3e-88006f24c167\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f" Oct 30 01:45:19.377130 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:19.377078 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f347ecd6-e2d7-43da-ba3e-88006f24c167-credential-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f\" (UID: \"f347ecd6-e2d7-43da-ba3e-88006f24c167\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f" Oct 30 01:45:19.377496 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:19.377249 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f347ecd6-e2d7-43da-ba3e-88006f24c167-workload-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f\" (UID: \"f347ecd6-e2d7-43da-ba3e-88006f24c167\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f" Oct 30 01:45:19.377649 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:19.377276 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f347ecd6-e2d7-43da-ba3e-88006f24c167-workload-certs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f\" (UID: \"f347ecd6-e2d7-43da-ba3e-88006f24c167\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f" Oct 30 01:45:19.377649 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:19.377287 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f347ecd6-e2d7-43da-ba3e-88006f24c167-istiod-ca-cert\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f\" (UID: \"f347ecd6-e2d7-43da-ba3e-88006f24c167\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f" Oct 30 01:45:19.379210 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:19.379188 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f347ecd6-e2d7-43da-ba3e-88006f24c167-istio-envoy\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f\" (UID: \"f347ecd6-e2d7-43da-ba3e-88006f24c167\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f" Oct 30 01:45:19.379482 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:19.379463 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f347ecd6-e2d7-43da-ba3e-88006f24c167-istio-podinfo\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f\" (UID: \"f347ecd6-e2d7-43da-ba3e-88006f24c167\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f" Oct 30 01:45:19.397142 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:19.397112 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f347ecd6-e2d7-43da-ba3e-88006f24c167-istio-token\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f\" (UID: \"f347ecd6-e2d7-43da-ba3e-88006f24c167\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f" Oct 30 01:45:19.397331 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:19.397314 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmprs\" (UniqueName: \"kubernetes.io/projected/f347ecd6-e2d7-43da-ba3e-88006f24c167-kube-api-access-jmprs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f\" (UID: \"f347ecd6-e2d7-43da-ba3e-88006f24c167\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f" Oct 30 01:45:19.506509 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:19.506423 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f" Oct 30 01:45:19.824790 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:19.824759 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f"] Oct 30 01:45:19.827986 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:45:19.827955 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf347ecd6_e2d7_43da_ba3e_88006f24c167.slice/crio-29527bd4e551b2d2dd86e3d7767cbb0c792f595fb4c5ff45e150f6b7ae87793d WatchSource:0}: Error finding container 29527bd4e551b2d2dd86e3d7767cbb0c792f595fb4c5ff45e150f6b7ae87793d: Status 404 returned error can't find the container with id 29527bd4e551b2d2dd86e3d7767cbb0c792f595fb4c5ff45e150f6b7ae87793d Oct 30 01:45:19.830947 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:19.830904 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:45:19.831054 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:19.831001 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:45:19.831054 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:19.831042 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:45:19.876171 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:19.876136 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f" event={"ID":"f347ecd6-e2d7-43da-ba3e-88006f24c167","Type":"ContainerStarted","Data":"29527bd4e551b2d2dd86e3d7767cbb0c792f595fb4c5ff45e150f6b7ae87793d"} Oct 30 01:45:20.883422 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:20.883387 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f" event={"ID":"f347ecd6-e2d7-43da-ba3e-88006f24c167","Type":"ContainerStarted","Data":"091691472dff64c611649bc7c4dfcbd1d44e80ec09d0e24a27c69311e45297f6"} Oct 30 01:45:20.974830 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:20.974773 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f" podStartSLOduration=1.974757689 podStartE2EDuration="1.974757689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:20.95975181 +0000 UTC m=+1644.538039147" watchObservedRunningTime="2026-10-30 01:45:20.974757689 +0000 UTC m=+1644.553045101" Oct 30 01:45:21.494152 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:45:21.494117 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:45:21.531663 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:21.531629 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f" Oct 30 01:45:21.533066 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:21.533037 6221 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 01:45:21.533213 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:21.533107 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f" podUID="f347ecd6-e2d7-43da-ba3e-88006f24c167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 01:45:22.510556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:22.507692 6221 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 01:45:22.510556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:22.507765 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f" podUID="f347ecd6-e2d7-43da-ba3e-88006f24c167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 01:45:23.517555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:23.515923 6221 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 01:45:23.517555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:23.516002 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f" podUID="f347ecd6-e2d7-43da-ba3e-88006f24c167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 01:45:23.753463 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:23.753427 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q"] Oct 30 01:45:23.757557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:23.757524 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" Oct 30 01:45:23.775274 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:23.775209 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-8f68ab-openshift-default-dockercfg-qzjzc\"" Oct 30 01:45:23.804493 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:23.804452 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q"] Oct 30 01:45:23.950557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:23.917449 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbbfc7ea-d855-4b77-b484-b0b316f40963-credential-socket\") pod \"e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q\" (UID: \"cbbfc7ea-d855-4b77-b484-b0b316f40963\") " pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" Oct 30 01:45:23.950958 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:23.950926 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbbfc7ea-d855-4b77-b484-b0b316f40963-istiod-ca-cert\") pod \"e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q\" (UID: \"cbbfc7ea-d855-4b77-b484-b0b316f40963\") " pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" Oct 30 01:45:23.951136 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:23.951115 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbbfc7ea-d855-4b77-b484-b0b316f40963-istio-data\") pod \"e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q\" (UID: \"cbbfc7ea-d855-4b77-b484-b0b316f40963\") " pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" Oct 30 01:45:23.951269 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:23.951252 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lzqt\" (UniqueName: \"kubernetes.io/projected/cbbfc7ea-d855-4b77-b484-b0b316f40963-kube-api-access-7lzqt\") pod \"e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q\" (UID: \"cbbfc7ea-d855-4b77-b484-b0b316f40963\") " pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" Oct 30 01:45:23.951431 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:23.951415 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbbfc7ea-d855-4b77-b484-b0b316f40963-workload-socket\") pod \"e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q\" (UID: \"cbbfc7ea-d855-4b77-b484-b0b316f40963\") " pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" Oct 30 01:45:24.002692 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:24.002632 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbbfc7ea-d855-4b77-b484-b0b316f40963-istio-podinfo\") pod \"e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q\" (UID: \"cbbfc7ea-d855-4b77-b484-b0b316f40963\") " pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" Oct 30 01:45:24.004557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:24.003085 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbbfc7ea-d855-4b77-b484-b0b316f40963-istio-envoy\") pod \"e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q\" (UID: \"cbbfc7ea-d855-4b77-b484-b0b316f40963\") " pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" Oct 30 01:45:24.030027 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:24.029852 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbbfc7ea-d855-4b77-b484-b0b316f40963-workload-certs\") pod \"e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q\" (UID: \"cbbfc7ea-d855-4b77-b484-b0b316f40963\") " pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" Oct 30 01:45:24.030405 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:24.030374 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbbfc7ea-d855-4b77-b484-b0b316f40963-istio-token\") pod \"e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q\" (UID: \"cbbfc7ea-d855-4b77-b484-b0b316f40963\") " pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" Oct 30 01:45:24.133553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:24.133343 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbbfc7ea-d855-4b77-b484-b0b316f40963-workload-socket\") pod \"e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q\" (UID: \"cbbfc7ea-d855-4b77-b484-b0b316f40963\") " pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" Oct 30 01:45:24.133553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:24.133418 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbbfc7ea-d855-4b77-b484-b0b316f40963-istio-podinfo\") pod \"e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q\" (UID: \"cbbfc7ea-d855-4b77-b484-b0b316f40963\") " pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" Oct 30 01:45:24.133553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:24.133492 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbbfc7ea-d855-4b77-b484-b0b316f40963-istio-envoy\") pod \"e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q\" (UID: \"cbbfc7ea-d855-4b77-b484-b0b316f40963\") " pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" Oct 30 01:45:24.133878 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:24.133577 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbbfc7ea-d855-4b77-b484-b0b316f40963-workload-certs\") pod \"e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q\" (UID: \"cbbfc7ea-d855-4b77-b484-b0b316f40963\") " pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" Oct 30 01:45:24.133878 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:24.133645 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbbfc7ea-d855-4b77-b484-b0b316f40963-istio-token\") pod \"e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q\" (UID: \"cbbfc7ea-d855-4b77-b484-b0b316f40963\") " pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" Oct 30 01:45:24.133878 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:24.133700 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbbfc7ea-d855-4b77-b484-b0b316f40963-credential-socket\") pod \"e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q\" (UID: \"cbbfc7ea-d855-4b77-b484-b0b316f40963\") " pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" Oct 30 01:45:24.133878 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:24.133769 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbbfc7ea-d855-4b77-b484-b0b316f40963-istiod-ca-cert\") pod \"e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q\" (UID: \"cbbfc7ea-d855-4b77-b484-b0b316f40963\") " pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" Oct 30 01:45:24.133878 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:24.133810 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbbfc7ea-d855-4b77-b484-b0b316f40963-istio-data\") pod \"e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q\" (UID: \"cbbfc7ea-d855-4b77-b484-b0b316f40963\") " pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" Oct 30 01:45:24.133878 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:24.133844 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lzqt\" (UniqueName: \"kubernetes.io/projected/cbbfc7ea-d855-4b77-b484-b0b316f40963-kube-api-access-7lzqt\") pod \"e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q\" (UID: \"cbbfc7ea-d855-4b77-b484-b0b316f40963\") " pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" Oct 30 01:45:24.141555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:24.134636 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbbfc7ea-d855-4b77-b484-b0b316f40963-workload-socket\") pod \"e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q\" (UID: \"cbbfc7ea-d855-4b77-b484-b0b316f40963\") " pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" Oct 30 01:45:24.141555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:24.141087 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbbfc7ea-d855-4b77-b484-b0b316f40963-istio-envoy\") pod \"e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q\" (UID: \"cbbfc7ea-d855-4b77-b484-b0b316f40963\") " pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" Oct 30 01:45:24.141555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:24.141347 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbbfc7ea-d855-4b77-b484-b0b316f40963-workload-certs\") pod \"e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q\" (UID: \"cbbfc7ea-d855-4b77-b484-b0b316f40963\") " pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" Oct 30 01:45:24.141916 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:24.141897 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbbfc7ea-d855-4b77-b484-b0b316f40963-istiod-ca-cert\") pod \"e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q\" (UID: \"cbbfc7ea-d855-4b77-b484-b0b316f40963\") " pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" Oct 30 01:45:24.142169 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:24.142145 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbbfc7ea-d855-4b77-b484-b0b316f40963-credential-socket\") pod \"e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q\" (UID: \"cbbfc7ea-d855-4b77-b484-b0b316f40963\") " pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" Oct 30 01:45:24.142327 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:24.142314 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbbfc7ea-d855-4b77-b484-b0b316f40963-istio-data\") pod \"e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q\" (UID: \"cbbfc7ea-d855-4b77-b484-b0b316f40963\") " pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" Oct 30 01:45:24.143022 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:24.143001 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbbfc7ea-d855-4b77-b484-b0b316f40963-istio-podinfo\") pod \"e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q\" (UID: \"cbbfc7ea-d855-4b77-b484-b0b316f40963\") " pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" Oct 30 01:45:24.154878 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:24.154845 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lzqt\" (UniqueName: \"kubernetes.io/projected/cbbfc7ea-d855-4b77-b484-b0b316f40963-kube-api-access-7lzqt\") pod \"e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q\" (UID: \"cbbfc7ea-d855-4b77-b484-b0b316f40963\") " pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" Oct 30 01:45:24.155325 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:24.155300 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbbfc7ea-d855-4b77-b484-b0b316f40963-istio-token\") pod \"e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q\" (UID: \"cbbfc7ea-d855-4b77-b484-b0b316f40963\") " pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" Oct 30 01:45:24.371701 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:24.371585 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" Oct 30 01:45:24.529554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:24.521966 6221 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 01:45:24.529554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:24.522027 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f" podUID="f347ecd6-e2d7-43da-ba3e-88006f24c167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 01:45:24.725940 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:24.725434 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q"] Oct 30 01:45:24.755578 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:45:24.736200 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbbfc7ea_d855_4b77_b484_b0b316f40963.slice/crio-c28c56adc9744b7920a9d2ab5cb9eadd5d38bc70f9fc811e5de60c55b8e0d815 WatchSource:0}: Error finding container c28c56adc9744b7920a9d2ab5cb9eadd5d38bc70f9fc811e5de60c55b8e0d815: Status 404 returned error can't find the container with id c28c56adc9744b7920a9d2ab5cb9eadd5d38bc70f9fc811e5de60c55b8e0d815 Oct 30 01:45:24.760436 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:24.760393 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:45:24.760548 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:24.760475 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:45:24.760548 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:24.760507 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:45:25.145857 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:25.142280 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" event={"ID":"cbbfc7ea-d855-4b77-b484-b0b316f40963","Type":"ContainerStarted","Data":"443dc49ae30b6ac2faf2336d59c3ed039cae65ec6c66df5f05ed79e88a85a8cc"} Oct 30 01:45:25.145857 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:25.142390 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" event={"ID":"cbbfc7ea-d855-4b77-b484-b0b316f40963","Type":"ContainerStarted","Data":"c28c56adc9744b7920a9d2ab5cb9eadd5d38bc70f9fc811e5de60c55b8e0d815"} Oct 30 01:45:25.195910 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:25.195857 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" podStartSLOduration=2.195843429 podStartE2EDuration="2.195843429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:25.194048561 +0000 UTC m=+1648.772335925" watchObservedRunningTime="2026-10-30 01:45:25.195843429 +0000 UTC m=+1648.774130765" Oct 30 01:45:25.375570 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:25.374863 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" Oct 30 01:45:25.382366 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:25.381895 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" Oct 30 01:45:25.531700 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:25.528931 6221 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 01:45:25.531700 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:25.528995 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f" podUID="f347ecd6-e2d7-43da-ba3e-88006f24c167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 01:45:26.147838 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:26.147801 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" Oct 30 01:45:26.148898 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:26.148877 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" Oct 30 01:45:26.506898 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:26.506816 6221 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body= Oct 30 01:45:26.506898 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:26.506883 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f" podUID="f347ecd6-e2d7-43da-ba3e-88006f24c167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" Oct 30 01:45:27.512134 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:27.512105 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f" Oct 30 01:45:27.512601 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:27.512570 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f" Oct 30 01:45:27.513345 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:27.513329 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f" Oct 30 01:45:32.067555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:32.066609 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk"] Oct 30 01:45:32.067555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:32.066898 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk" podUID="cd15071b-edcd-4d7f-8309-c02a3281dcb5" containerName="istio-proxy" containerID="cri-o://15c8a196d36428f6ab70ca0f10fb46afaca0f513cefe290c916400a7ee054b5b" gracePeriod=30 Oct 30 01:45:32.077864 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:32.077387 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4"] Oct 30 01:45:32.077864 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:32.077667 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4" podUID="39be085b-e445-416b-93c6-52de82dc6a83" containerName="istio-proxy" containerID="cri-o://4608e02e79850ed8a57ec28dbe23da1936d60464caab397aaf2d771cd5710af3" gracePeriod=30 Oct 30 01:45:32.098845 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:32.098810 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b"] Oct 30 01:45:32.099055 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:32.099032 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b" podUID="f33783d9-0aa2-4089-878d-ffc93fded2c8" containerName="istio-proxy" containerID="cri-o://834aefe8b3df55015510da5c1c63f2ab13cab3ad189129004881e6deef12f73c" gracePeriod=30 Oct 30 01:45:32.102774 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:32.102743 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh"] Oct 30 01:45:32.103017 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:32.102982 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" podUID="c9bddd88-6e7d-47fe-99b1-11c8208f3fef" containerName="istio-proxy" containerID="cri-o://3bcc7fc6e2ffd25b8cea2a3e14f29a57c274edd7dd09862e3613195eb373ff79" gracePeriod=30 Oct 30 01:45:35.340706 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.340670 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9"] Oct 30 01:45:35.346073 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.346050 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" Oct 30 01:45:35.355500 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.355470 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj"] Oct 30 01:45:35.359861 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.359838 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj" Oct 30 01:45:35.366651 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.366040 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-a3a2ac-openshift-default-dockercfg-mwwtj\"" Oct 30 01:45:35.370457 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.370430 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9"] Oct 30 01:45:35.382288 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.382263 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj"] Oct 30 01:45:35.462556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.460792 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48557095-c2dc-40cb-8ee3-8ee71567c61c-istio-envoy\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9\" (UID: \"48557095-c2dc-40cb-8ee3-8ee71567c61c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" Oct 30 01:45:35.462556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.460843 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-workload-certs\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj\" (UID: \"3e8eba77-efd7-4b5c-8093-4af09c7dce5c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj" Oct 30 01:45:35.462556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.460870 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-istio-envoy\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj\" (UID: \"3e8eba77-efd7-4b5c-8093-4af09c7dce5c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj" Oct 30 01:45:35.462556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.460900 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-istio-token\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj\" (UID: \"3e8eba77-efd7-4b5c-8093-4af09c7dce5c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj" Oct 30 01:45:35.462556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.460933 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4tsv\" (UniqueName: \"kubernetes.io/projected/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-kube-api-access-k4tsv\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj\" (UID: \"3e8eba77-efd7-4b5c-8093-4af09c7dce5c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj" Oct 30 01:45:35.462556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.461013 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-istio-podinfo\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj\" (UID: \"3e8eba77-efd7-4b5c-8093-4af09c7dce5c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj" Oct 30 01:45:35.462556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.461053 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-istio-data\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj\" (UID: \"3e8eba77-efd7-4b5c-8093-4af09c7dce5c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj" Oct 30 01:45:35.462556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.461085 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jbts\" (UniqueName: \"kubernetes.io/projected/48557095-c2dc-40cb-8ee3-8ee71567c61c-kube-api-access-4jbts\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9\" (UID: \"48557095-c2dc-40cb-8ee3-8ee71567c61c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" Oct 30 01:45:35.462556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.461149 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48557095-c2dc-40cb-8ee3-8ee71567c61c-workload-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9\" (UID: \"48557095-c2dc-40cb-8ee3-8ee71567c61c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" Oct 30 01:45:35.462556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.461196 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48557095-c2dc-40cb-8ee3-8ee71567c61c-istio-data\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9\" (UID: \"48557095-c2dc-40cb-8ee3-8ee71567c61c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" Oct 30 01:45:35.462556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.461235 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-workload-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj\" (UID: \"3e8eba77-efd7-4b5c-8093-4af09c7dce5c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj" Oct 30 01:45:35.462556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.461264 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-credential-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj\" (UID: \"3e8eba77-efd7-4b5c-8093-4af09c7dce5c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj" Oct 30 01:45:35.462556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.461301 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48557095-c2dc-40cb-8ee3-8ee71567c61c-workload-certs\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9\" (UID: \"48557095-c2dc-40cb-8ee3-8ee71567c61c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" Oct 30 01:45:35.462556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.461334 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48557095-c2dc-40cb-8ee3-8ee71567c61c-istiod-ca-cert\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9\" (UID: \"48557095-c2dc-40cb-8ee3-8ee71567c61c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" Oct 30 01:45:35.462556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.461367 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48557095-c2dc-40cb-8ee3-8ee71567c61c-istio-token\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9\" (UID: \"48557095-c2dc-40cb-8ee3-8ee71567c61c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" Oct 30 01:45:35.462556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.461437 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48557095-c2dc-40cb-8ee3-8ee71567c61c-istio-podinfo\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9\" (UID: \"48557095-c2dc-40cb-8ee3-8ee71567c61c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" Oct 30 01:45:35.462556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.461513 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-istiod-ca-cert\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj\" (UID: \"3e8eba77-efd7-4b5c-8093-4af09c7dce5c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj" Oct 30 01:45:35.462556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.461587 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48557095-c2dc-40cb-8ee3-8ee71567c61c-credential-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9\" (UID: \"48557095-c2dc-40cb-8ee3-8ee71567c61c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" Oct 30 01:45:35.562757 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.562720 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48557095-c2dc-40cb-8ee3-8ee71567c61c-workload-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9\" (UID: \"48557095-c2dc-40cb-8ee3-8ee71567c61c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" Oct 30 01:45:35.562957 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.562769 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48557095-c2dc-40cb-8ee3-8ee71567c61c-istio-data\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9\" (UID: \"48557095-c2dc-40cb-8ee3-8ee71567c61c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" Oct 30 01:45:35.562957 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.562934 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-workload-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj\" (UID: \"3e8eba77-efd7-4b5c-8093-4af09c7dce5c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj" Oct 30 01:45:35.563093 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.562987 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-credential-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj\" (UID: \"3e8eba77-efd7-4b5c-8093-4af09c7dce5c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj" Oct 30 01:45:35.563093 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.563026 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48557095-c2dc-40cb-8ee3-8ee71567c61c-workload-certs\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9\" (UID: \"48557095-c2dc-40cb-8ee3-8ee71567c61c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" Oct 30 01:45:35.563093 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.563061 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48557095-c2dc-40cb-8ee3-8ee71567c61c-istiod-ca-cert\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9\" (UID: \"48557095-c2dc-40cb-8ee3-8ee71567c61c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" Oct 30 01:45:35.563260 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.563095 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48557095-c2dc-40cb-8ee3-8ee71567c61c-istio-token\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9\" (UID: \"48557095-c2dc-40cb-8ee3-8ee71567c61c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" Oct 30 01:45:35.563260 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.563100 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48557095-c2dc-40cb-8ee3-8ee71567c61c-istio-data\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9\" (UID: \"48557095-c2dc-40cb-8ee3-8ee71567c61c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" Oct 30 01:45:35.563260 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.563166 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48557095-c2dc-40cb-8ee3-8ee71567c61c-istio-podinfo\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9\" (UID: \"48557095-c2dc-40cb-8ee3-8ee71567c61c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" Oct 30 01:45:35.563260 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.563173 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48557095-c2dc-40cb-8ee3-8ee71567c61c-workload-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9\" (UID: \"48557095-c2dc-40cb-8ee3-8ee71567c61c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" Oct 30 01:45:35.563469 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.563269 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-istiod-ca-cert\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj\" (UID: \"3e8eba77-efd7-4b5c-8093-4af09c7dce5c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj" Oct 30 01:45:35.563469 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.563329 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48557095-c2dc-40cb-8ee3-8ee71567c61c-workload-certs\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9\" (UID: \"48557095-c2dc-40cb-8ee3-8ee71567c61c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" Oct 30 01:45:35.563469 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.563334 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-workload-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj\" (UID: \"3e8eba77-efd7-4b5c-8093-4af09c7dce5c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj" Oct 30 01:45:35.563469 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.563328 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48557095-c2dc-40cb-8ee3-8ee71567c61c-credential-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9\" (UID: \"48557095-c2dc-40cb-8ee3-8ee71567c61c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" Oct 30 01:45:35.563469 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.563401 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-credential-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj\" (UID: \"3e8eba77-efd7-4b5c-8093-4af09c7dce5c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj" Oct 30 01:45:35.563735 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.563552 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48557095-c2dc-40cb-8ee3-8ee71567c61c-istio-envoy\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9\" (UID: \"48557095-c2dc-40cb-8ee3-8ee71567c61c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" Oct 30 01:45:35.563735 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.563586 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-workload-certs\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj\" (UID: \"3e8eba77-efd7-4b5c-8093-4af09c7dce5c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj" Oct 30 01:45:35.563735 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.563596 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48557095-c2dc-40cb-8ee3-8ee71567c61c-credential-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9\" (UID: \"48557095-c2dc-40cb-8ee3-8ee71567c61c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" Oct 30 01:45:35.563735 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.563608 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-istio-envoy\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj\" (UID: \"3e8eba77-efd7-4b5c-8093-4af09c7dce5c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj" Oct 30 01:45:35.563735 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.563657 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-istio-token\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj\" (UID: \"3e8eba77-efd7-4b5c-8093-4af09c7dce5c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj" Oct 30 01:45:35.563735 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.563681 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4tsv\" (UniqueName: \"kubernetes.io/projected/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-kube-api-access-k4tsv\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj\" (UID: \"3e8eba77-efd7-4b5c-8093-4af09c7dce5c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj" Oct 30 01:45:35.563735 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.563714 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-istio-podinfo\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj\" (UID: \"3e8eba77-efd7-4b5c-8093-4af09c7dce5c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj" Oct 30 01:45:35.564094 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.563741 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-istio-data\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj\" (UID: \"3e8eba77-efd7-4b5c-8093-4af09c7dce5c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj" Oct 30 01:45:35.564094 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.563762 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jbts\" (UniqueName: \"kubernetes.io/projected/48557095-c2dc-40cb-8ee3-8ee71567c61c-kube-api-access-4jbts\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9\" (UID: \"48557095-c2dc-40cb-8ee3-8ee71567c61c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" Oct 30 01:45:35.564094 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.563904 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48557095-c2dc-40cb-8ee3-8ee71567c61c-istiod-ca-cert\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9\" (UID: \"48557095-c2dc-40cb-8ee3-8ee71567c61c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" Oct 30 01:45:35.564094 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.564028 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-istiod-ca-cert\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj\" (UID: \"3e8eba77-efd7-4b5c-8093-4af09c7dce5c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj" Oct 30 01:45:35.564342 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.564172 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-workload-certs\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj\" (UID: \"3e8eba77-efd7-4b5c-8093-4af09c7dce5c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj" Oct 30 01:45:35.564342 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.564194 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-istio-data\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj\" (UID: \"3e8eba77-efd7-4b5c-8093-4af09c7dce5c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj" Oct 30 01:45:35.565856 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.565836 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48557095-c2dc-40cb-8ee3-8ee71567c61c-istio-envoy\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9\" (UID: \"48557095-c2dc-40cb-8ee3-8ee71567c61c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" Oct 30 01:45:35.565963 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.565909 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-istio-envoy\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj\" (UID: \"3e8eba77-efd7-4b5c-8093-4af09c7dce5c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj" Oct 30 01:45:35.566389 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.566364 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48557095-c2dc-40cb-8ee3-8ee71567c61c-istio-podinfo\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9\" (UID: \"48557095-c2dc-40cb-8ee3-8ee71567c61c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" Oct 30 01:45:35.566497 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.566479 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-istio-podinfo\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj\" (UID: \"3e8eba77-efd7-4b5c-8093-4af09c7dce5c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj" Oct 30 01:45:35.588735 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.588707 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-istio-token\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj\" (UID: \"3e8eba77-efd7-4b5c-8093-4af09c7dce5c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj" Oct 30 01:45:35.588894 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.588711 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jbts\" (UniqueName: \"kubernetes.io/projected/48557095-c2dc-40cb-8ee3-8ee71567c61c-kube-api-access-4jbts\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9\" (UID: \"48557095-c2dc-40cb-8ee3-8ee71567c61c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" Oct 30 01:45:35.589291 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.589264 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48557095-c2dc-40cb-8ee3-8ee71567c61c-istio-token\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9\" (UID: \"48557095-c2dc-40cb-8ee3-8ee71567c61c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" Oct 30 01:45:35.589636 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.589618 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4tsv\" (UniqueName: \"kubernetes.io/projected/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-kube-api-access-k4tsv\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj\" (UID: \"3e8eba77-efd7-4b5c-8093-4af09c7dce5c\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj" Oct 30 01:45:35.663101 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.663013 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" Oct 30 01:45:35.674765 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.674737 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj" Oct 30 01:45:35.999271 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:35.999247 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9"] Oct 30 01:45:36.001959 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:36.001930 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj"] Oct 30 01:45:36.002122 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:45:36.001998 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48557095_c2dc_40cb_8ee3_8ee71567c61c.slice/crio-ab50521be4f8cc8990d422aa7453f027e6088e5bd94e3d3619c43a2ba8bbf22a WatchSource:0}: Error finding container ab50521be4f8cc8990d422aa7453f027e6088e5bd94e3d3619c43a2ba8bbf22a: Status 404 returned error can't find the container with id ab50521be4f8cc8990d422aa7453f027e6088e5bd94e3d3619c43a2ba8bbf22a Oct 30 01:45:36.004506 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:36.004479 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:45:36.004583 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:36.004554 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:45:36.004628 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:36.004584 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:45:36.013804 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:45:36.013778 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e8eba77_efd7_4b5c_8093_4af09c7dce5c.slice/crio-679852c94cc1535af3f3486998ec5ca1b8ebeaf4f17cc17bdca8652510c18d39 WatchSource:0}: Error finding container 679852c94cc1535af3f3486998ec5ca1b8ebeaf4f17cc17bdca8652510c18d39: Status 404 returned error can't find the container with id 679852c94cc1535af3f3486998ec5ca1b8ebeaf4f17cc17bdca8652510c18d39 Oct 30 01:45:36.016247 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:36.016215 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:45:36.016348 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:36.016277 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:45:36.016348 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:36.016314 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:45:36.219825 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:36.219788 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:45:36.256402 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:36.256309 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj" event={"ID":"3e8eba77-efd7-4b5c-8093-4af09c7dce5c","Type":"ContainerStarted","Data":"cc04c43519df2931e1e23a15cc336d7e5f5780a01a26544100645dc3f197d73e"} Oct 30 01:45:36.256402 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:36.256372 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj" event={"ID":"3e8eba77-efd7-4b5c-8093-4af09c7dce5c","Type":"ContainerStarted","Data":"679852c94cc1535af3f3486998ec5ca1b8ebeaf4f17cc17bdca8652510c18d39"} Oct 30 01:45:36.258233 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:36.258199 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" event={"ID":"48557095-c2dc-40cb-8ee3-8ee71567c61c","Type":"ContainerStarted","Data":"02b28bf6299d286f6ec9a31d31008878b76c5e36c7bafa1122a21fa124e1219f"} Oct 30 01:45:36.258375 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:36.258246 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" event={"ID":"48557095-c2dc-40cb-8ee3-8ee71567c61c","Type":"ContainerStarted","Data":"ab50521be4f8cc8990d422aa7453f027e6088e5bd94e3d3619c43a2ba8bbf22a"} Oct 30 01:45:36.299001 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:36.298930 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj" podStartSLOduration=1.298911392 podStartE2EDuration="1.298911392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:36.296495541 +0000 UTC m=+1659.874782878" watchObservedRunningTime="2026-10-30 01:45:36.298911392 +0000 UTC m=+1659.877198729" Oct 30 01:45:36.342089 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:36.342033 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" podStartSLOduration=1.342016601 podStartE2EDuration="1.342016601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:36.338843032 +0000 UTC m=+1659.917130389" watchObservedRunningTime="2026-10-30 01:45:36.342016601 +0000 UTC m=+1659.920303937" Oct 30 01:45:36.663888 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:36.663814 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" Oct 30 01:45:36.675247 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:36.675216 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj" Oct 30 01:45:37.239086 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:37.238740 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-njbt4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 01:45:37.239086 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:37.238799 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4" podUID="39be085b-e445-416b-93c6-52de82dc6a83" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 01:45:37.339840 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:37.334648 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-qb7jk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 01:45:37.339840 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:37.334707 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk" podUID="cd15071b-edcd-4d7f-8309-c02a3281dcb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 01:45:37.377932 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:37.377900 6221 generic.go:356] "Generic (PLEG): container finished" podID="cd15071b-edcd-4d7f-8309-c02a3281dcb5" containerID="15c8a196d36428f6ab70ca0f10fb46afaca0f513cefe290c916400a7ee054b5b" exitCode=0 Oct 30 01:45:37.395769 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:37.383840 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk" event={"ID":"cd15071b-edcd-4d7f-8309-c02a3281dcb5","Type":"ContainerDied","Data":"15c8a196d36428f6ab70ca0f10fb46afaca0f513cefe290c916400a7ee054b5b"} Oct 30 01:45:37.671227 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:37.668740 6221 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 01:45:37.671227 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:37.668805 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" podUID="48557095-c2dc-40cb-8ee3-8ee71567c61c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 01:45:37.694804 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:37.694770 6221 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 01:45:37.695065 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:37.695039 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj" podUID="3e8eba77-efd7-4b5c-8093-4af09c7dce5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 01:45:37.989107 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:37.988728 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk" Oct 30 01:45:38.001250 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:37.999980 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd15071b-edcd-4d7f-8309-c02a3281dcb5-istio-token\" (UniqueName: \"kubernetes.io/projected/cd15071b-edcd-4d7f-8309-c02a3281dcb5-istio-token\") pod \"cd15071b-edcd-4d7f-8309-c02a3281dcb5\" (UID: \"cd15071b-edcd-4d7f-8309-c02a3281dcb5\") " Oct 30 01:45:38.001250 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.000097 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd15071b-edcd-4d7f-8309-c02a3281dcb5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd15071b-edcd-4d7f-8309-c02a3281dcb5-istiod-ca-cert\") pod \"cd15071b-edcd-4d7f-8309-c02a3281dcb5\" (UID: \"cd15071b-edcd-4d7f-8309-c02a3281dcb5\") " Oct 30 01:45:38.001250 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.000153 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cd15071b-edcd-4d7f-8309-c02a3281dcb5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd15071b-edcd-4d7f-8309-c02a3281dcb5-istio-podinfo\") pod \"cd15071b-edcd-4d7f-8309-c02a3281dcb5\" (UID: \"cd15071b-edcd-4d7f-8309-c02a3281dcb5\") " Oct 30 01:45:38.001250 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.000191 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd15071b-edcd-4d7f-8309-c02a3281dcb5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd15071b-edcd-4d7f-8309-c02a3281dcb5-istio-envoy\") pod \"cd15071b-edcd-4d7f-8309-c02a3281dcb5\" (UID: \"cd15071b-edcd-4d7f-8309-c02a3281dcb5\") " Oct 30 01:45:38.001250 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.000255 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd15071b-edcd-4d7f-8309-c02a3281dcb5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd15071b-edcd-4d7f-8309-c02a3281dcb5-istio-data\") pod \"cd15071b-edcd-4d7f-8309-c02a3281dcb5\" (UID: \"cd15071b-edcd-4d7f-8309-c02a3281dcb5\") " Oct 30 01:45:38.001250 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.000308 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd15071b-edcd-4d7f-8309-c02a3281dcb5-kube-api-access-27md5\" (UniqueName: \"kubernetes.io/projected/cd15071b-edcd-4d7f-8309-c02a3281dcb5-kube-api-access-27md5\") pod \"cd15071b-edcd-4d7f-8309-c02a3281dcb5\" (UID: \"cd15071b-edcd-4d7f-8309-c02a3281dcb5\") " Oct 30 01:45:38.001250 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.000381 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd15071b-edcd-4d7f-8309-c02a3281dcb5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd15071b-edcd-4d7f-8309-c02a3281dcb5-workload-certs\") pod \"cd15071b-edcd-4d7f-8309-c02a3281dcb5\" (UID: \"cd15071b-edcd-4d7f-8309-c02a3281dcb5\") " Oct 30 01:45:38.001250 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.000417 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd15071b-edcd-4d7f-8309-c02a3281dcb5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd15071b-edcd-4d7f-8309-c02a3281dcb5-credential-socket\") pod \"cd15071b-edcd-4d7f-8309-c02a3281dcb5\" (UID: \"cd15071b-edcd-4d7f-8309-c02a3281dcb5\") " Oct 30 01:45:38.001250 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.000458 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd15071b-edcd-4d7f-8309-c02a3281dcb5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd15071b-edcd-4d7f-8309-c02a3281dcb5-workload-socket\") pod \"cd15071b-edcd-4d7f-8309-c02a3281dcb5\" (UID: \"cd15071b-edcd-4d7f-8309-c02a3281dcb5\") " Oct 30 01:45:38.003223 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.002890 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd15071b-edcd-4d7f-8309-c02a3281dcb5-workload-socket" pod "cd15071b-edcd-4d7f-8309-c02a3281dcb5" (UID: "cd15071b-edcd-4d7f-8309-c02a3281dcb5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:45:38.003413 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.003337 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd15071b-edcd-4d7f-8309-c02a3281dcb5-istio-data" pod "cd15071b-edcd-4d7f-8309-c02a3281dcb5" (UID: "cd15071b-edcd-4d7f-8309-c02a3281dcb5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:45:38.004296 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.003690 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd15071b-edcd-4d7f-8309-c02a3281dcb5-istiod-ca-cert" pod "cd15071b-edcd-4d7f-8309-c02a3281dcb5" (UID: "cd15071b-edcd-4d7f-8309-c02a3281dcb5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:45:38.004404 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.004318 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd15071b-edcd-4d7f-8309-c02a3281dcb5-workload-certs" pod "cd15071b-edcd-4d7f-8309-c02a3281dcb5" (UID: "cd15071b-edcd-4d7f-8309-c02a3281dcb5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:45:38.004884 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.004862 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd15071b-edcd-4d7f-8309-c02a3281dcb5-credential-socket" pod "cd15071b-edcd-4d7f-8309-c02a3281dcb5" (UID: "cd15071b-edcd-4d7f-8309-c02a3281dcb5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:45:38.029923 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.029664 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cd15071b-edcd-4d7f-8309-c02a3281dcb5-istio-podinfo" pod "cd15071b-edcd-4d7f-8309-c02a3281dcb5" (UID: "cd15071b-edcd-4d7f-8309-c02a3281dcb5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:45:38.030714 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.030670 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd15071b-edcd-4d7f-8309-c02a3281dcb5-istio-token" pod "cd15071b-edcd-4d7f-8309-c02a3281dcb5" (UID: "cd15071b-edcd-4d7f-8309-c02a3281dcb5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:45:38.054202 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.053672 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd15071b-edcd-4d7f-8309-c02a3281dcb5-istio-envoy" pod "cd15071b-edcd-4d7f-8309-c02a3281dcb5" (UID: "cd15071b-edcd-4d7f-8309-c02a3281dcb5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:45:38.070604 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.058040 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd15071b-edcd-4d7f-8309-c02a3281dcb5-kube-api-access-27md5" pod "cd15071b-edcd-4d7f-8309-c02a3281dcb5" (UID: "cd15071b-edcd-4d7f-8309-c02a3281dcb5"). InnerVolumeSpecName "kube-api-access-27md5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:45:38.104562 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.104121 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cd15071b-edcd-4d7f-8309-c02a3281dcb5-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:45:38.104562 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.104150 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cd15071b-edcd-4d7f-8309-c02a3281dcb5-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:45:38.104562 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.104159 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27md5\" (UniqueName: \"kubernetes.io/projected/cd15071b-edcd-4d7f-8309-c02a3281dcb5-kube-api-access-27md5\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:45:38.104562 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.104168 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cd15071b-edcd-4d7f-8309-c02a3281dcb5-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:45:38.104562 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.104177 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd15071b-edcd-4d7f-8309-c02a3281dcb5-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:45:38.104562 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.104185 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cd15071b-edcd-4d7f-8309-c02a3281dcb5-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:45:38.104562 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.104193 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cd15071b-edcd-4d7f-8309-c02a3281dcb5-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:45:38.104562 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.104202 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cd15071b-edcd-4d7f-8309-c02a3281dcb5-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:45:38.104562 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.104210 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cd15071b-edcd-4d7f-8309-c02a3281dcb5-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:45:38.499419 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.499246 6221 generic.go:356] "Generic (PLEG): container finished" podID="39be085b-e445-416b-93c6-52de82dc6a83" containerID="4608e02e79850ed8a57ec28dbe23da1936d60464caab397aaf2d771cd5710af3" exitCode=0 Oct 30 01:45:38.499419 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.499346 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4" event={"ID":"39be085b-e445-416b-93c6-52de82dc6a83","Type":"ContainerDied","Data":"4608e02e79850ed8a57ec28dbe23da1936d60464caab397aaf2d771cd5710af3"} Oct 30 01:45:38.502037 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.501817 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk" event={"ID":"cd15071b-edcd-4d7f-8309-c02a3281dcb5","Type":"ContainerDied","Data":"5675e86a59121cafd0a55c731a9ec79dc20398c640fd0436b8f6febb499b9535"} Oct 30 01:45:38.502037 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.501871 6221 scope.go:122] "RemoveContainer" containerID="15c8a196d36428f6ab70ca0f10fb46afaca0f513cefe290c916400a7ee054b5b" Oct 30 01:45:38.504572 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.502242 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk" Oct 30 01:45:38.548925 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.548860 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" Oct 30 01:45:38.557553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.553687 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b" Oct 30 01:45:38.565163 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.564320 6221 generic.go:356] "Generic (PLEG): container finished" podID="f33783d9-0aa2-4089-878d-ffc93fded2c8" containerID="834aefe8b3df55015510da5c1c63f2ab13cab3ad189129004881e6deef12f73c" exitCode=0 Oct 30 01:45:38.567575 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.566385 6221 generic.go:356] "Generic (PLEG): container finished" podID="c9bddd88-6e7d-47fe-99b1-11c8208f3fef" containerID="3bcc7fc6e2ffd25b8cea2a3e14f29a57c274edd7dd09862e3613195eb373ff79" exitCode=0 Oct 30 01:45:38.567575 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.567278 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" Oct 30 01:45:38.595489 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.595456 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4" Oct 30 01:45:38.598452 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.596989 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b" event={"ID":"f33783d9-0aa2-4089-878d-ffc93fded2c8","Type":"ContainerDied","Data":"834aefe8b3df55015510da5c1c63f2ab13cab3ad189129004881e6deef12f73c"} Oct 30 01:45:38.598452 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.597046 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b" event={"ID":"f33783d9-0aa2-4089-878d-ffc93fded2c8","Type":"ContainerDied","Data":"ed404d7c83c9f2507744a3a8568326b6fc3a6623d821881e938eab3e391b4ab6"} Oct 30 01:45:38.598452 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.597072 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" event={"ID":"c9bddd88-6e7d-47fe-99b1-11c8208f3fef","Type":"ContainerDied","Data":"3bcc7fc6e2ffd25b8cea2a3e14f29a57c274edd7dd09862e3613195eb373ff79"} Oct 30 01:45:38.598452 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.597100 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh" event={"ID":"c9bddd88-6e7d-47fe-99b1-11c8208f3fef","Type":"ContainerDied","Data":"3075f2de3899365594f657cbc184ca1da78069872cbf293f8eedeccf0b1e0c85"} Oct 30 01:45:38.598452 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.597163 6221 scope.go:122] "RemoveContainer" containerID="834aefe8b3df55015510da5c1c63f2ab13cab3ad189129004881e6deef12f73c" Oct 30 01:45:38.694985 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.694950 6221 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 01:45:38.695214 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.695185 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" podUID="48557095-c2dc-40cb-8ee3-8ee71567c61c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 01:45:38.705571 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.698900 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f33783d9-0aa2-4089-878d-ffc93fded2c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f33783d9-0aa2-4089-878d-ffc93fded2c8-istio-podinfo\") pod \"f33783d9-0aa2-4089-878d-ffc93fded2c8\" (UID: \"f33783d9-0aa2-4089-878d-ffc93fded2c8\") " Oct 30 01:45:38.705802 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.705782 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39be085b-e445-416b-93c6-52de82dc6a83-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39be085b-e445-416b-93c6-52de82dc6a83-istio-envoy\") pod \"39be085b-e445-416b-93c6-52de82dc6a83\" (UID: \"39be085b-e445-416b-93c6-52de82dc6a83\") " Oct 30 01:45:38.705929 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.705916 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39be085b-e445-416b-93c6-52de82dc6a83-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39be085b-e445-416b-93c6-52de82dc6a83-istio-podinfo\") pod \"39be085b-e445-416b-93c6-52de82dc6a83\" (UID: \"39be085b-e445-416b-93c6-52de82dc6a83\") " Oct 30 01:45:38.764929 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.704836 6221 scope.go:122] "RemoveContainer" containerID="834aefe8b3df55015510da5c1c63f2ab13cab3ad189129004881e6deef12f73c" Oct 30 01:45:38.764929 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.706060 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f33783d9-0aa2-4089-878d-ffc93fded2c8-istio-token\" (UniqueName: \"kubernetes.io/projected/f33783d9-0aa2-4089-878d-ffc93fded2c8-istio-token\") pod \"f33783d9-0aa2-4089-878d-ffc93fded2c8\" (UID: \"f33783d9-0aa2-4089-878d-ffc93fded2c8\") " Oct 30 01:45:38.764929 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.706143 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-istio-envoy\") pod \"c9bddd88-6e7d-47fe-99b1-11c8208f3fef\" (UID: \"c9bddd88-6e7d-47fe-99b1-11c8208f3fef\") " Oct 30 01:45:38.764929 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.700812 6221 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 01:45:38.764929 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.706178 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f33783d9-0aa2-4089-878d-ffc93fded2c8-kube-api-access-fgzsj\" (UniqueName: \"kubernetes.io/projected/f33783d9-0aa2-4089-878d-ffc93fded2c8-kube-api-access-fgzsj\") pod \"f33783d9-0aa2-4089-878d-ffc93fded2c8\" (UID: \"f33783d9-0aa2-4089-878d-ffc93fded2c8\") " Oct 30 01:45:38.764929 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.706199 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj" podUID="3e8eba77-efd7-4b5c-8093-4af09c7dce5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 01:45:38.764929 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.703635 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f33783d9-0aa2-4089-878d-ffc93fded2c8-istio-podinfo" pod "f33783d9-0aa2-4089-878d-ffc93fded2c8" (UID: "f33783d9-0aa2-4089-878d-ffc93fded2c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:45:38.764929 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.706238 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f33783d9-0aa2-4089-878d-ffc93fded2c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f33783d9-0aa2-4089-878d-ffc93fded2c8-istio-data\") pod \"f33783d9-0aa2-4089-878d-ffc93fded2c8\" (UID: \"f33783d9-0aa2-4089-878d-ffc93fded2c8\") " Oct 30 01:45:38.764929 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.706303 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-istio-token\" (UniqueName: \"kubernetes.io/projected/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-istio-token\") pod \"c9bddd88-6e7d-47fe-99b1-11c8208f3fef\" (UID: \"c9bddd88-6e7d-47fe-99b1-11c8208f3fef\") " Oct 30 01:45:38.764929 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.706346 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39be085b-e445-416b-93c6-52de82dc6a83-kube-api-access-2ztpr\" (UniqueName: \"kubernetes.io/projected/39be085b-e445-416b-93c6-52de82dc6a83-kube-api-access-2ztpr\") pod \"39be085b-e445-416b-93c6-52de82dc6a83\" (UID: \"39be085b-e445-416b-93c6-52de82dc6a83\") " Oct 30 01:45:38.764929 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.706378 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39be085b-e445-416b-93c6-52de82dc6a83-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39be085b-e445-416b-93c6-52de82dc6a83-istiod-ca-cert\") pod \"39be085b-e445-416b-93c6-52de82dc6a83\" (UID: \"39be085b-e445-416b-93c6-52de82dc6a83\") " Oct 30 01:45:38.764929 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.706413 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f33783d9-0aa2-4089-878d-ffc93fded2c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f33783d9-0aa2-4089-878d-ffc93fded2c8-istiod-ca-cert\") pod \"f33783d9-0aa2-4089-878d-ffc93fded2c8\" (UID: \"f33783d9-0aa2-4089-878d-ffc93fded2c8\") " Oct 30 01:45:38.764929 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.706443 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f33783d9-0aa2-4089-878d-ffc93fded2c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f33783d9-0aa2-4089-878d-ffc93fded2c8-workload-socket\") pod \"f33783d9-0aa2-4089-878d-ffc93fded2c8\" (UID: \"f33783d9-0aa2-4089-878d-ffc93fded2c8\") " Oct 30 01:45:38.764929 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.706472 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f33783d9-0aa2-4089-878d-ffc93fded2c8-istio-data" pod "f33783d9-0aa2-4089-878d-ffc93fded2c8" (UID: "f33783d9-0aa2-4089-878d-ffc93fded2c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:45:38.764929 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.706488 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f33783d9-0aa2-4089-878d-ffc93fded2c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f33783d9-0aa2-4089-878d-ffc93fded2c8-workload-certs\") pod \"f33783d9-0aa2-4089-878d-ffc93fded2c8\" (UID: \"f33783d9-0aa2-4089-878d-ffc93fded2c8\") " Oct 30 01:45:38.764929 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.706526 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39be085b-e445-416b-93c6-52de82dc6a83-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39be085b-e445-416b-93c6-52de82dc6a83-istio-data\") pod \"39be085b-e445-416b-93c6-52de82dc6a83\" (UID: \"39be085b-e445-416b-93c6-52de82dc6a83\") " Oct 30 01:45:38.764929 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.706629 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-istio-podinfo\") pod \"c9bddd88-6e7d-47fe-99b1-11c8208f3fef\" (UID: \"c9bddd88-6e7d-47fe-99b1-11c8208f3fef\") " Oct 30 01:45:38.764929 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.706682 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f33783d9-0aa2-4089-878d-ffc93fded2c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f33783d9-0aa2-4089-878d-ffc93fded2c8-istio-envoy\") pod \"f33783d9-0aa2-4089-878d-ffc93fded2c8\" (UID: \"f33783d9-0aa2-4089-878d-ffc93fded2c8\") " Oct 30 01:45:38.764929 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.706713 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f33783d9-0aa2-4089-878d-ffc93fded2c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f33783d9-0aa2-4089-878d-ffc93fded2c8-credential-socket\") pod \"f33783d9-0aa2-4089-878d-ffc93fded2c8\" (UID: \"f33783d9-0aa2-4089-878d-ffc93fded2c8\") " Oct 30 01:45:38.764929 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.706747 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39be085b-e445-416b-93c6-52de82dc6a83-istio-token\" (UniqueName: \"kubernetes.io/projected/39be085b-e445-416b-93c6-52de82dc6a83-istio-token\") pod \"39be085b-e445-416b-93c6-52de82dc6a83\" (UID: \"39be085b-e445-416b-93c6-52de82dc6a83\") " Oct 30 01:45:38.764929 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.706807 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39be085b-e445-416b-93c6-52de82dc6a83-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39be085b-e445-416b-93c6-52de82dc6a83-workload-certs\") pod \"39be085b-e445-416b-93c6-52de82dc6a83\" (UID: \"39be085b-e445-416b-93c6-52de82dc6a83\") " Oct 30 01:45:38.764929 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.706894 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39be085b-e445-416b-93c6-52de82dc6a83-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39be085b-e445-416b-93c6-52de82dc6a83-workload-socket\") pod \"39be085b-e445-416b-93c6-52de82dc6a83\" (UID: \"39be085b-e445-416b-93c6-52de82dc6a83\") " Oct 30 01:45:38.764929 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.706933 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39be085b-e445-416b-93c6-52de82dc6a83-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39be085b-e445-416b-93c6-52de82dc6a83-credential-socket\") pod \"39be085b-e445-416b-93c6-52de82dc6a83\" (UID: \"39be085b-e445-416b-93c6-52de82dc6a83\") " Oct 30 01:45:38.764929 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.707000 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-kube-api-access-49lp4\" (UniqueName: \"kubernetes.io/projected/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-kube-api-access-49lp4\") pod \"c9bddd88-6e7d-47fe-99b1-11c8208f3fef\" (UID: \"c9bddd88-6e7d-47fe-99b1-11c8208f3fef\") " Oct 30 01:45:38.764929 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.707075 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-istio-data\") pod \"c9bddd88-6e7d-47fe-99b1-11c8208f3fef\" (UID: \"c9bddd88-6e7d-47fe-99b1-11c8208f3fef\") " Oct 30 01:45:38.764929 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.707106 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-istiod-ca-cert\") pod \"c9bddd88-6e7d-47fe-99b1-11c8208f3fef\" (UID: \"c9bddd88-6e7d-47fe-99b1-11c8208f3fef\") " Oct 30 01:45:38.764929 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.707145 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-workload-certs\") pod \"c9bddd88-6e7d-47fe-99b1-11c8208f3fef\" (UID: \"c9bddd88-6e7d-47fe-99b1-11c8208f3fef\") " Oct 30 01:45:38.764929 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.707203 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-workload-socket\") pod \"c9bddd88-6e7d-47fe-99b1-11c8208f3fef\" (UID: \"c9bddd88-6e7d-47fe-99b1-11c8208f3fef\") " Oct 30 01:45:38.764929 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.707249 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-credential-socket\") pod \"c9bddd88-6e7d-47fe-99b1-11c8208f3fef\" (UID: \"c9bddd88-6e7d-47fe-99b1-11c8208f3fef\") " Oct 30 01:45:38.764929 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.708085 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f33783d9-0aa2-4089-878d-ffc93fded2c8-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:45:38.764929 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.708109 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f33783d9-0aa2-4089-878d-ffc93fded2c8-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:45:38.764929 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.749324 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39be085b-e445-416b-93c6-52de82dc6a83-credential-socket" pod "39be085b-e445-416b-93c6-52de82dc6a83" (UID: "39be085b-e445-416b-93c6-52de82dc6a83"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:45:38.804565 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.797675 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39be085b-e445-416b-93c6-52de82dc6a83-istiod-ca-cert" pod "39be085b-e445-416b-93c6-52de82dc6a83" (UID: "39be085b-e445-416b-93c6-52de82dc6a83"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:45:38.804565 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.798091 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f33783d9-0aa2-4089-878d-ffc93fded2c8-istiod-ca-cert" pod "f33783d9-0aa2-4089-878d-ffc93fded2c8" (UID: "f33783d9-0aa2-4089-878d-ffc93fded2c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:45:38.804565 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.798342 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f33783d9-0aa2-4089-878d-ffc93fded2c8-workload-socket" pod "f33783d9-0aa2-4089-878d-ffc93fded2c8" (UID: "f33783d9-0aa2-4089-878d-ffc93fded2c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:45:38.804565 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.798589 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f33783d9-0aa2-4089-878d-ffc93fded2c8-workload-certs" pod "f33783d9-0aa2-4089-878d-ffc93fded2c8" (UID: "f33783d9-0aa2-4089-878d-ffc93fded2c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:45:38.804565 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.798787 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39be085b-e445-416b-93c6-52de82dc6a83-istio-data" pod "39be085b-e445-416b-93c6-52de82dc6a83" (UID: "39be085b-e445-416b-93c6-52de82dc6a83"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:45:38.804565 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:45:38.798922 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"834aefe8b3df55015510da5c1c63f2ab13cab3ad189129004881e6deef12f73c\": container with ID starting with 834aefe8b3df55015510da5c1c63f2ab13cab3ad189129004881e6deef12f73c not found: ID does not exist" containerID="834aefe8b3df55015510da5c1c63f2ab13cab3ad189129004881e6deef12f73c" Oct 30 01:45:38.804565 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.798962 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"834aefe8b3df55015510da5c1c63f2ab13cab3ad189129004881e6deef12f73c"} err="failed to get container status \"834aefe8b3df55015510da5c1c63f2ab13cab3ad189129004881e6deef12f73c\": rpc error: code = NotFound desc = could not find container \"834aefe8b3df55015510da5c1c63f2ab13cab3ad189129004881e6deef12f73c\": container with ID starting with 834aefe8b3df55015510da5c1c63f2ab13cab3ad189129004881e6deef12f73c not found: ID does not exist" Oct 30 01:45:38.804565 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.798990 6221 scope.go:122] "RemoveContainer" containerID="3bcc7fc6e2ffd25b8cea2a3e14f29a57c274edd7dd09862e3613195eb373ff79" Oct 30 01:45:38.810732 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.810695 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk"] Oct 30 01:45:38.885854 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.885718 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39be085b-e445-416b-93c6-52de82dc6a83-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:45:38.885854 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.885751 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39be085b-e445-416b-93c6-52de82dc6a83-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:45:38.885854 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.885768 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f33783d9-0aa2-4089-878d-ffc93fded2c8-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:45:38.885854 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.885783 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f33783d9-0aa2-4089-878d-ffc93fded2c8-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:45:38.885854 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.885797 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f33783d9-0aa2-4089-878d-ffc93fded2c8-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:45:38.885854 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.885811 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39be085b-e445-416b-93c6-52de82dc6a83-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:45:38.886882 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.886221 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-qb7jk"] Oct 30 01:45:38.901868 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.888011 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-workload-certs" pod "c9bddd88-6e7d-47fe-99b1-11c8208f3fef" (UID: "c9bddd88-6e7d-47fe-99b1-11c8208f3fef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:45:38.901868 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.890104 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-workload-socket" pod "c9bddd88-6e7d-47fe-99b1-11c8208f3fef" (UID: "c9bddd88-6e7d-47fe-99b1-11c8208f3fef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:45:38.901868 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.890510 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39be085b-e445-416b-93c6-52de82dc6a83-workload-certs" pod "39be085b-e445-416b-93c6-52de82dc6a83" (UID: "39be085b-e445-416b-93c6-52de82dc6a83"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:45:38.901868 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.890689 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-istio-podinfo" pod "c9bddd88-6e7d-47fe-99b1-11c8208f3fef" (UID: "c9bddd88-6e7d-47fe-99b1-11c8208f3fef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:45:38.901868 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.890729 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f33783d9-0aa2-4089-878d-ffc93fded2c8-istio-token" pod "f33783d9-0aa2-4089-878d-ffc93fded2c8" (UID: "f33783d9-0aa2-4089-878d-ffc93fded2c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:45:38.901868 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.890757 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39be085b-e445-416b-93c6-52de82dc6a83-workload-socket" pod "39be085b-e445-416b-93c6-52de82dc6a83" (UID: "39be085b-e445-416b-93c6-52de82dc6a83"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:45:38.901868 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.891580 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39be085b-e445-416b-93c6-52de82dc6a83-istio-podinfo" pod "39be085b-e445-416b-93c6-52de82dc6a83" (UID: "39be085b-e445-416b-93c6-52de82dc6a83"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:45:38.902341 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.902166 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-istio-data" pod "c9bddd88-6e7d-47fe-99b1-11c8208f3fef" (UID: "c9bddd88-6e7d-47fe-99b1-11c8208f3fef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:45:38.902408 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.902374 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-credential-socket" pod "c9bddd88-6e7d-47fe-99b1-11c8208f3fef" (UID: "c9bddd88-6e7d-47fe-99b1-11c8208f3fef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:45:38.902551 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.902517 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-istiod-ca-cert" pod "c9bddd88-6e7d-47fe-99b1-11c8208f3fef" (UID: "c9bddd88-6e7d-47fe-99b1-11c8208f3fef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:45:38.903423 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.903400 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39be085b-e445-416b-93c6-52de82dc6a83-istio-token" pod "39be085b-e445-416b-93c6-52de82dc6a83" (UID: "39be085b-e445-416b-93c6-52de82dc6a83"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:45:38.904014 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.903986 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39be085b-e445-416b-93c6-52de82dc6a83-istio-envoy" pod "39be085b-e445-416b-93c6-52de82dc6a83" (UID: "39be085b-e445-416b-93c6-52de82dc6a83"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:45:38.904014 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.903987 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f33783d9-0aa2-4089-878d-ffc93fded2c8-istio-envoy" pod "f33783d9-0aa2-4089-878d-ffc93fded2c8" (UID: "f33783d9-0aa2-4089-878d-ffc93fded2c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:45:38.904466 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.904444 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f33783d9-0aa2-4089-878d-ffc93fded2c8-kube-api-access-fgzsj" pod "f33783d9-0aa2-4089-878d-ffc93fded2c8" (UID: "f33783d9-0aa2-4089-878d-ffc93fded2c8"). InnerVolumeSpecName "kube-api-access-fgzsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:45:38.904827 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.904793 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-kube-api-access-49lp4" pod "c9bddd88-6e7d-47fe-99b1-11c8208f3fef" (UID: "c9bddd88-6e7d-47fe-99b1-11c8208f3fef"). InnerVolumeSpecName "kube-api-access-49lp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:45:38.905147 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.905127 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-istio-envoy" pod "c9bddd88-6e7d-47fe-99b1-11c8208f3fef" (UID: "c9bddd88-6e7d-47fe-99b1-11c8208f3fef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:45:38.905405 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.905374 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-istio-token" pod "c9bddd88-6e7d-47fe-99b1-11c8208f3fef" (UID: "c9bddd88-6e7d-47fe-99b1-11c8208f3fef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:45:38.905513 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.905498 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39be085b-e445-416b-93c6-52de82dc6a83-kube-api-access-2ztpr" pod "39be085b-e445-416b-93c6-52de82dc6a83" (UID: "39be085b-e445-416b-93c6-52de82dc6a83"). InnerVolumeSpecName "kube-api-access-2ztpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:45:38.916780 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.915794 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f33783d9-0aa2-4089-878d-ffc93fded2c8-credential-socket" pod "f33783d9-0aa2-4089-878d-ffc93fded2c8" (UID: "f33783d9-0aa2-4089-878d-ffc93fded2c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:45:38.991883 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.987521 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:45:38.991883 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.990611 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:45:38.991883 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.990633 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:45:38.991883 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.990649 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39be085b-e445-416b-93c6-52de82dc6a83-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:45:38.991883 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.990665 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39be085b-e445-416b-93c6-52de82dc6a83-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:45:38.991883 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.990678 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f33783d9-0aa2-4089-878d-ffc93fded2c8-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:45:38.991883 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.990691 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:45:38.991883 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.990704 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgzsj\" (UniqueName: \"kubernetes.io/projected/f33783d9-0aa2-4089-878d-ffc93fded2c8-kube-api-access-fgzsj\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:45:38.991883 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.990719 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:45:38.991883 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.990734 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ztpr\" (UniqueName: \"kubernetes.io/projected/39be085b-e445-416b-93c6-52de82dc6a83-kube-api-access-2ztpr\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:45:38.991883 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.990749 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:45:38.991883 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.990763 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f33783d9-0aa2-4089-878d-ffc93fded2c8-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:45:38.991883 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.990776 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f33783d9-0aa2-4089-878d-ffc93fded2c8-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:45:38.991883 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.990790 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39be085b-e445-416b-93c6-52de82dc6a83-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:45:38.991883 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.990804 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39be085b-e445-416b-93c6-52de82dc6a83-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:45:38.991883 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.990817 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39be085b-e445-416b-93c6-52de82dc6a83-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:45:38.991883 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.990831 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49lp4\" (UniqueName: \"kubernetes.io/projected/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-kube-api-access-49lp4\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:45:38.991883 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.990847 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:45:38.991883 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:38.990860 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9bddd88-6e7d-47fe-99b1-11c8208f3fef-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:45:39.049926 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:39.049048 6221 scope.go:122] "RemoveContainer" containerID="3bcc7fc6e2ffd25b8cea2a3e14f29a57c274edd7dd09862e3613195eb373ff79" Oct 30 01:45:39.049926 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:45:39.049553 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bcc7fc6e2ffd25b8cea2a3e14f29a57c274edd7dd09862e3613195eb373ff79\": container with ID starting with 3bcc7fc6e2ffd25b8cea2a3e14f29a57c274edd7dd09862e3613195eb373ff79 not found: ID does not exist" containerID="3bcc7fc6e2ffd25b8cea2a3e14f29a57c274edd7dd09862e3613195eb373ff79" Oct 30 01:45:39.049926 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:39.049590 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bcc7fc6e2ffd25b8cea2a3e14f29a57c274edd7dd09862e3613195eb373ff79"} err="failed to get container status \"3bcc7fc6e2ffd25b8cea2a3e14f29a57c274edd7dd09862e3613195eb373ff79\": rpc error: code = NotFound desc = could not find container \"3bcc7fc6e2ffd25b8cea2a3e14f29a57c274edd7dd09862e3613195eb373ff79\": container with ID starting with 3bcc7fc6e2ffd25b8cea2a3e14f29a57c274edd7dd09862e3613195eb373ff79 not found: ID does not exist" Oct 30 01:45:39.272410 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:39.272296 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh"] Oct 30 01:45:39.277402 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:39.277356 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-26hnh"] Oct 30 01:45:39.671579 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:39.669678 6221 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 01:45:39.671579 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:39.669740 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" podUID="48557095-c2dc-40cb-8ee3-8ee71567c61c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 01:45:39.672905 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:39.672871 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4" event={"ID":"39be085b-e445-416b-93c6-52de82dc6a83","Type":"ContainerDied","Data":"ddf38bb0bc42a40e57e9b58ef54a87fe14b0ce7bccf65f7872bf6f9ed52015ad"} Oct 30 01:45:39.673017 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:39.672949 6221 scope.go:122] "RemoveContainer" containerID="4608e02e79850ed8a57ec28dbe23da1936d60464caab397aaf2d771cd5710af3" Oct 30 01:45:39.673351 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:39.673330 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4" Oct 30 01:45:39.675141 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:39.675119 6221 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 01:45:39.675211 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:39.675187 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj" podUID="3e8eba77-efd7-4b5c-8093-4af09c7dce5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 01:45:39.678685 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:39.678651 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b" Oct 30 01:45:39.764307 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:39.764271 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4"] Oct 30 01:45:39.775856 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:39.775825 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-njbt4"] Oct 30 01:45:39.802006 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:39.801968 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b"] Oct 30 01:45:39.820556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:39.809333 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-wvm5b"] Oct 30 01:45:40.562301 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:40.562254 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39be085b-e445-416b-93c6-52de82dc6a83" path="/var/lib/kubelet/pods/39be085b-e445-416b-93c6-52de82dc6a83/volumes" Oct 30 01:45:40.563059 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:40.563037 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9bddd88-6e7d-47fe-99b1-11c8208f3fef" path="/var/lib/kubelet/pods/c9bddd88-6e7d-47fe-99b1-11c8208f3fef/volumes" Oct 30 01:45:40.563713 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:40.563695 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd15071b-edcd-4d7f-8309-c02a3281dcb5" path="/var/lib/kubelet/pods/cd15071b-edcd-4d7f-8309-c02a3281dcb5/volumes" Oct 30 01:45:40.564473 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:40.564457 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f33783d9-0aa2-4089-878d-ffc93fded2c8" path="/var/lib/kubelet/pods/f33783d9-0aa2-4089-878d-ffc93fded2c8/volumes" Oct 30 01:45:40.663864 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:40.663716 6221 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 01:45:40.663864 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:40.663778 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" podUID="48557095-c2dc-40cb-8ee3-8ee71567c61c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 01:45:40.675271 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:40.675240 6221 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 01:45:40.675692 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:40.675291 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj" podUID="3e8eba77-efd7-4b5c-8093-4af09c7dce5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 01:45:41.664972 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:41.664932 6221 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 01:45:41.665279 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:41.665251 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" podUID="48557095-c2dc-40cb-8ee3-8ee71567c61c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 01:45:41.675620 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:41.675579 6221 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 01:45:41.675990 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:41.675645 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj" podUID="3e8eba77-efd7-4b5c-8093-4af09c7dce5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 01:45:42.664327 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:42.664261 6221 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 01:45:42.664518 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:42.664329 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" podUID="48557095-c2dc-40cb-8ee3-8ee71567c61c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 01:45:42.675722 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:42.675683 6221 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 01:45:42.676200 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:42.675740 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj" podUID="3e8eba77-efd7-4b5c-8093-4af09c7dce5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 01:45:42.734440 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:42.734396 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-8f68ab-5bf445cc7d-9b2mq"] Oct 30 01:45:42.735275 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:42.735235 6221 cpu_manager.go:414] "containerMap: removing container" podUID="39be085b-e445-416b-93c6-52de82dc6a83" containerName="istio-proxy" Oct 30 01:45:42.735397 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:42.735264 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39be085b-e445-416b-93c6-52de82dc6a83" containerName="istio-proxy" Oct 30 01:45:42.735397 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:42.735311 6221 cpu_manager.go:414] "containerMap: removing container" podUID="f33783d9-0aa2-4089-878d-ffc93fded2c8" containerName="istio-proxy" Oct 30 01:45:42.735397 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:42.735323 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f33783d9-0aa2-4089-878d-ffc93fded2c8" containerName="istio-proxy" Oct 30 01:45:42.735397 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:42.735349 6221 cpu_manager.go:414] "containerMap: removing container" podUID="cd15071b-edcd-4d7f-8309-c02a3281dcb5" containerName="istio-proxy" Oct 30 01:45:42.735397 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:42.735358 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd15071b-edcd-4d7f-8309-c02a3281dcb5" containerName="istio-proxy" Oct 30 01:45:42.735585 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:42.735402 6221 cpu_manager.go:414] "containerMap: removing container" podUID="c9bddd88-6e7d-47fe-99b1-11c8208f3fef" containerName="istio-proxy" Oct 30 01:45:42.735585 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:42.735410 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9bddd88-6e7d-47fe-99b1-11c8208f3fef" containerName="istio-proxy" Oct 30 01:45:42.735585 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:42.735559 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="39be085b-e445-416b-93c6-52de82dc6a83" containerName="istio-proxy" Oct 30 01:45:42.735585 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:42.735578 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="f33783d9-0aa2-4089-878d-ffc93fded2c8" containerName="istio-proxy" Oct 30 01:45:42.735700 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:42.735591 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd15071b-edcd-4d7f-8309-c02a3281dcb5" containerName="istio-proxy" Oct 30 01:45:42.735700 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:42.735609 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9bddd88-6e7d-47fe-99b1-11c8208f3fef" containerName="istio-proxy" Oct 30 01:45:42.741249 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:42.741229 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8f68ab-5bf445cc7d-9b2mq" Oct 30 01:45:42.755498 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:42.755474 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-8f68ab-5bf445cc7d-9b2mq"] Oct 30 01:45:42.755802 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:42.755783 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-8f68ab-serving-cert\"" Oct 30 01:45:42.802950 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:42.802918 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64vv2\" (UniqueName: \"kubernetes.io/projected/9abd0232-8d93-4dfa-ad61-7b01ed80eba8-kube-api-access-64vv2\") pod \"maas-api-e2e-praxis-b-8f68ab-5bf445cc7d-9b2mq\" (UID: \"9abd0232-8d93-4dfa-ad61-7b01ed80eba8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8f68ab-5bf445cc7d-9b2mq" Oct 30 01:45:42.803139 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:42.803078 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9abd0232-8d93-4dfa-ad61-7b01ed80eba8-metrics-certs\") pod \"maas-api-e2e-praxis-b-8f68ab-5bf445cc7d-9b2mq\" (UID: \"9abd0232-8d93-4dfa-ad61-7b01ed80eba8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8f68ab-5bf445cc7d-9b2mq" Oct 30 01:45:42.803139 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:42.803116 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9abd0232-8d93-4dfa-ad61-7b01ed80eba8-maas-api-tls\") pod \"maas-api-e2e-praxis-b-8f68ab-5bf445cc7d-9b2mq\" (UID: \"9abd0232-8d93-4dfa-ad61-7b01ed80eba8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8f68ab-5bf445cc7d-9b2mq" Oct 30 01:45:42.803230 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:42.803148 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9abd0232-8d93-4dfa-ad61-7b01ed80eba8-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-8f68ab-5bf445cc7d-9b2mq\" (UID: \"9abd0232-8d93-4dfa-ad61-7b01ed80eba8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8f68ab-5bf445cc7d-9b2mq" Oct 30 01:45:42.904076 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:42.904040 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9abd0232-8d93-4dfa-ad61-7b01ed80eba8-metrics-certs\") pod \"maas-api-e2e-praxis-b-8f68ab-5bf445cc7d-9b2mq\" (UID: \"9abd0232-8d93-4dfa-ad61-7b01ed80eba8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8f68ab-5bf445cc7d-9b2mq" Oct 30 01:45:42.904250 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:42.904088 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9abd0232-8d93-4dfa-ad61-7b01ed80eba8-maas-api-tls\") pod \"maas-api-e2e-praxis-b-8f68ab-5bf445cc7d-9b2mq\" (UID: \"9abd0232-8d93-4dfa-ad61-7b01ed80eba8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8f68ab-5bf445cc7d-9b2mq" Oct 30 01:45:42.904250 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:42.904112 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9abd0232-8d93-4dfa-ad61-7b01ed80eba8-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-8f68ab-5bf445cc7d-9b2mq\" (UID: \"9abd0232-8d93-4dfa-ad61-7b01ed80eba8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8f68ab-5bf445cc7d-9b2mq" Oct 30 01:45:42.904250 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:42.904203 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64vv2\" (UniqueName: \"kubernetes.io/projected/9abd0232-8d93-4dfa-ad61-7b01ed80eba8-kube-api-access-64vv2\") pod \"maas-api-e2e-praxis-b-8f68ab-5bf445cc7d-9b2mq\" (UID: \"9abd0232-8d93-4dfa-ad61-7b01ed80eba8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8f68ab-5bf445cc7d-9b2mq" Oct 30 01:45:42.905089 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:42.905057 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9abd0232-8d93-4dfa-ad61-7b01ed80eba8-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-8f68ab-5bf445cc7d-9b2mq\" (UID: \"9abd0232-8d93-4dfa-ad61-7b01ed80eba8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8f68ab-5bf445cc7d-9b2mq" Oct 30 01:45:42.906664 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:42.906638 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9abd0232-8d93-4dfa-ad61-7b01ed80eba8-maas-api-tls\") pod \"maas-api-e2e-praxis-b-8f68ab-5bf445cc7d-9b2mq\" (UID: \"9abd0232-8d93-4dfa-ad61-7b01ed80eba8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8f68ab-5bf445cc7d-9b2mq" Oct 30 01:45:42.906779 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:42.906688 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9abd0232-8d93-4dfa-ad61-7b01ed80eba8-metrics-certs\") pod \"maas-api-e2e-praxis-b-8f68ab-5bf445cc7d-9b2mq\" (UID: \"9abd0232-8d93-4dfa-ad61-7b01ed80eba8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8f68ab-5bf445cc7d-9b2mq" Oct 30 01:45:42.923711 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:42.923556 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64vv2\" (UniqueName: \"kubernetes.io/projected/9abd0232-8d93-4dfa-ad61-7b01ed80eba8-kube-api-access-64vv2\") pod \"maas-api-e2e-praxis-b-8f68ab-5bf445cc7d-9b2mq\" (UID: \"9abd0232-8d93-4dfa-ad61-7b01ed80eba8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8f68ab-5bf445cc7d-9b2mq" Oct 30 01:45:43.060034 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.059953 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8f68ab-5bf445cc7d-9b2mq" Oct 30 01:45:43.143384 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.140041 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf"] Oct 30 01:45:43.156152 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.156115 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh"] Oct 30 01:45:43.156303 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.156232 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf" Oct 30 01:45:43.165008 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.164985 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh" Oct 30 01:45:43.201411 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.198640 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf"] Oct 30 01:45:43.205865 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.205835 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh"] Oct 30 01:45:43.209791 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.209760 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b8071ca-a7f0-47a5-aabf-32148fa0463a-workload-certs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf\" (UID: \"7b8071ca-a7f0-47a5-aabf-32148fa0463a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf" Oct 30 01:45:43.209895 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.209826 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-istio-token\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh\" (UID: \"59eaf4dd-3059-4fa3-bac8-93fc2b4d518a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh" Oct 30 01:45:43.209895 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.209863 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzbhc\" (UniqueName: \"kubernetes.io/projected/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-kube-api-access-xzbhc\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh\" (UID: \"59eaf4dd-3059-4fa3-bac8-93fc2b4d518a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh" Oct 30 01:45:43.209980 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.209902 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b8071ca-a7f0-47a5-aabf-32148fa0463a-istio-data\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf\" (UID: \"7b8071ca-a7f0-47a5-aabf-32148fa0463a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf" Oct 30 01:45:43.209980 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.209928 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-istiod-ca-cert\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh\" (UID: \"59eaf4dd-3059-4fa3-bac8-93fc2b4d518a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh" Oct 30 01:45:43.210045 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.209991 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b8071ca-a7f0-47a5-aabf-32148fa0463a-istio-token\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf\" (UID: \"7b8071ca-a7f0-47a5-aabf-32148fa0463a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf" Oct 30 01:45:43.210045 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.210018 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b8071ca-a7f0-47a5-aabf-32148fa0463a-credential-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf\" (UID: \"7b8071ca-a7f0-47a5-aabf-32148fa0463a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf" Oct 30 01:45:43.210106 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.210061 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-istio-data\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh\" (UID: \"59eaf4dd-3059-4fa3-bac8-93fc2b4d518a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh" Oct 30 01:45:43.210106 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.210091 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b8071ca-a7f0-47a5-aabf-32148fa0463a-istio-envoy\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf\" (UID: \"7b8071ca-a7f0-47a5-aabf-32148fa0463a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf" Oct 30 01:45:43.210167 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.210117 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b8071ca-a7f0-47a5-aabf-32148fa0463a-istiod-ca-cert\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf\" (UID: \"7b8071ca-a7f0-47a5-aabf-32148fa0463a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf" Oct 30 01:45:43.210202 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.210162 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-istio-envoy\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh\" (UID: \"59eaf4dd-3059-4fa3-bac8-93fc2b4d518a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh" Oct 30 01:45:43.210257 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.210208 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-credential-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh\" (UID: \"59eaf4dd-3059-4fa3-bac8-93fc2b4d518a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh" Oct 30 01:45:43.210257 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.210241 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-istio-podinfo\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh\" (UID: \"59eaf4dd-3059-4fa3-bac8-93fc2b4d518a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh" Oct 30 01:45:43.210354 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.210288 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzg6f\" (UniqueName: \"kubernetes.io/projected/7b8071ca-a7f0-47a5-aabf-32148fa0463a-kube-api-access-jzg6f\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf\" (UID: \"7b8071ca-a7f0-47a5-aabf-32148fa0463a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf" Oct 30 01:45:43.210354 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.210320 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-workload-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh\" (UID: \"59eaf4dd-3059-4fa3-bac8-93fc2b4d518a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh" Oct 30 01:45:43.210354 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.210346 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-workload-certs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh\" (UID: \"59eaf4dd-3059-4fa3-bac8-93fc2b4d518a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh" Oct 30 01:45:43.210638 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.210617 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b8071ca-a7f0-47a5-aabf-32148fa0463a-workload-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf\" (UID: \"7b8071ca-a7f0-47a5-aabf-32148fa0463a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf" Oct 30 01:45:43.210774 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.210741 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b8071ca-a7f0-47a5-aabf-32148fa0463a-istio-podinfo\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf\" (UID: \"7b8071ca-a7f0-47a5-aabf-32148fa0463a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf" Oct 30 01:45:43.311987 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.311952 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b8071ca-a7f0-47a5-aabf-32148fa0463a-istio-podinfo\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf\" (UID: \"7b8071ca-a7f0-47a5-aabf-32148fa0463a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf" Oct 30 01:45:43.312152 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.312001 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b8071ca-a7f0-47a5-aabf-32148fa0463a-workload-certs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf\" (UID: \"7b8071ca-a7f0-47a5-aabf-32148fa0463a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf" Oct 30 01:45:43.312152 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.312041 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-istio-token\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh\" (UID: \"59eaf4dd-3059-4fa3-bac8-93fc2b4d518a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh" Oct 30 01:45:43.312152 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.312065 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzbhc\" (UniqueName: \"kubernetes.io/projected/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-kube-api-access-xzbhc\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh\" (UID: \"59eaf4dd-3059-4fa3-bac8-93fc2b4d518a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh" Oct 30 01:45:43.312152 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.312097 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b8071ca-a7f0-47a5-aabf-32148fa0463a-istio-data\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf\" (UID: \"7b8071ca-a7f0-47a5-aabf-32148fa0463a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf" Oct 30 01:45:43.312152 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.312116 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-istiod-ca-cert\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh\" (UID: \"59eaf4dd-3059-4fa3-bac8-93fc2b4d518a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh" Oct 30 01:45:43.312449 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.312182 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b8071ca-a7f0-47a5-aabf-32148fa0463a-istio-token\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf\" (UID: \"7b8071ca-a7f0-47a5-aabf-32148fa0463a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf" Oct 30 01:45:43.312449 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.312213 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b8071ca-a7f0-47a5-aabf-32148fa0463a-credential-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf\" (UID: \"7b8071ca-a7f0-47a5-aabf-32148fa0463a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf" Oct 30 01:45:43.312449 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.312254 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-istio-data\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh\" (UID: \"59eaf4dd-3059-4fa3-bac8-93fc2b4d518a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh" Oct 30 01:45:43.312449 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.312285 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b8071ca-a7f0-47a5-aabf-32148fa0463a-istio-envoy\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf\" (UID: \"7b8071ca-a7f0-47a5-aabf-32148fa0463a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf" Oct 30 01:45:43.312449 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.312311 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b8071ca-a7f0-47a5-aabf-32148fa0463a-istiod-ca-cert\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf\" (UID: \"7b8071ca-a7f0-47a5-aabf-32148fa0463a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf" Oct 30 01:45:43.312449 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.312373 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-istio-envoy\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh\" (UID: \"59eaf4dd-3059-4fa3-bac8-93fc2b4d518a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh" Oct 30 01:45:43.312800 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.312456 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-credential-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh\" (UID: \"59eaf4dd-3059-4fa3-bac8-93fc2b4d518a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh" Oct 30 01:45:43.312800 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.312495 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b8071ca-a7f0-47a5-aabf-32148fa0463a-workload-certs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf\" (UID: \"7b8071ca-a7f0-47a5-aabf-32148fa0463a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf" Oct 30 01:45:43.312800 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.312510 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-istio-podinfo\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh\" (UID: \"59eaf4dd-3059-4fa3-bac8-93fc2b4d518a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh" Oct 30 01:45:43.312800 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.312601 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzg6f\" (UniqueName: \"kubernetes.io/projected/7b8071ca-a7f0-47a5-aabf-32148fa0463a-kube-api-access-jzg6f\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf\" (UID: \"7b8071ca-a7f0-47a5-aabf-32148fa0463a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf" Oct 30 01:45:43.312800 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.312640 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-workload-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh\" (UID: \"59eaf4dd-3059-4fa3-bac8-93fc2b4d518a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh" Oct 30 01:45:43.312800 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.312665 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-workload-certs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh\" (UID: \"59eaf4dd-3059-4fa3-bac8-93fc2b4d518a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh" Oct 30 01:45:43.312800 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.312715 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b8071ca-a7f0-47a5-aabf-32148fa0463a-workload-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf\" (UID: \"7b8071ca-a7f0-47a5-aabf-32148fa0463a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf" Oct 30 01:45:43.313141 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.313104 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-istio-data\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh\" (UID: \"59eaf4dd-3059-4fa3-bac8-93fc2b4d518a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh" Oct 30 01:45:43.313562 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.313334 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-istiod-ca-cert\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh\" (UID: \"59eaf4dd-3059-4fa3-bac8-93fc2b4d518a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh" Oct 30 01:45:43.313562 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.313334 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b8071ca-a7f0-47a5-aabf-32148fa0463a-istiod-ca-cert\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf\" (UID: \"7b8071ca-a7f0-47a5-aabf-32148fa0463a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf" Oct 30 01:45:43.313562 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.313446 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b8071ca-a7f0-47a5-aabf-32148fa0463a-istio-data\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf\" (UID: \"7b8071ca-a7f0-47a5-aabf-32148fa0463a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf" Oct 30 01:45:43.313562 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.313509 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b8071ca-a7f0-47a5-aabf-32148fa0463a-workload-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf\" (UID: \"7b8071ca-a7f0-47a5-aabf-32148fa0463a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf" Oct 30 01:45:43.313849 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.313679 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-credential-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh\" (UID: \"59eaf4dd-3059-4fa3-bac8-93fc2b4d518a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh" Oct 30 01:45:43.313849 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.313755 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-workload-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh\" (UID: \"59eaf4dd-3059-4fa3-bac8-93fc2b4d518a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh" Oct 30 01:45:43.313963 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.313948 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-workload-certs\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh\" (UID: \"59eaf4dd-3059-4fa3-bac8-93fc2b4d518a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh" Oct 30 01:45:43.314033 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.314009 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b8071ca-a7f0-47a5-aabf-32148fa0463a-credential-socket\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf\" (UID: \"7b8071ca-a7f0-47a5-aabf-32148fa0463a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf" Oct 30 01:45:43.315713 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.315646 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-istio-envoy\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh\" (UID: \"59eaf4dd-3059-4fa3-bac8-93fc2b4d518a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh" Oct 30 01:45:43.315713 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.315652 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b8071ca-a7f0-47a5-aabf-32148fa0463a-istio-envoy\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf\" (UID: \"7b8071ca-a7f0-47a5-aabf-32148fa0463a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf" Oct 30 01:45:43.316182 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.316160 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b8071ca-a7f0-47a5-aabf-32148fa0463a-istio-podinfo\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf\" (UID: \"7b8071ca-a7f0-47a5-aabf-32148fa0463a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf" Oct 30 01:45:43.316480 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.316456 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-istio-podinfo\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh\" (UID: \"59eaf4dd-3059-4fa3-bac8-93fc2b4d518a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh" Oct 30 01:45:43.333249 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.333222 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b8071ca-a7f0-47a5-aabf-32148fa0463a-istio-token\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf\" (UID: \"7b8071ca-a7f0-47a5-aabf-32148fa0463a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf" Oct 30 01:45:43.333915 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.333887 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-istio-token\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh\" (UID: \"59eaf4dd-3059-4fa3-bac8-93fc2b4d518a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh" Oct 30 01:45:43.335672 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.335649 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzg6f\" (UniqueName: \"kubernetes.io/projected/7b8071ca-a7f0-47a5-aabf-32148fa0463a-kube-api-access-jzg6f\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf\" (UID: \"7b8071ca-a7f0-47a5-aabf-32148fa0463a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf" Oct 30 01:45:43.335972 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.335954 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzbhc\" (UniqueName: \"kubernetes.io/projected/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-kube-api-access-xzbhc\") pod \"e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh\" (UID: \"59eaf4dd-3059-4fa3-bac8-93fc2b4d518a\") " pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh" Oct 30 01:45:43.367634 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.367401 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-8f68ab-5bf445cc7d-9b2mq"] Oct 30 01:45:43.370605 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:45:43.370576 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9abd0232_8d93_4dfa_ad61_7b01ed80eba8.slice/crio-a3d49d4d23ac30fddc1913199df3cdbf01a27ac0065c193ffa3d9b695456da3c WatchSource:0}: Error finding container a3d49d4d23ac30fddc1913199df3cdbf01a27ac0065c193ffa3d9b695456da3c: Status 404 returned error can't find the container with id a3d49d4d23ac30fddc1913199df3cdbf01a27ac0065c193ffa3d9b695456da3c Oct 30 01:45:43.498420 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.498316 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh" Oct 30 01:45:43.511726 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.511675 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf" Oct 30 01:45:43.740859 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.735479 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8f68ab-5bf445cc7d-9b2mq" event={"ID":"9abd0232-8d93-4dfa-ad61-7b01ed80eba8","Type":"ContainerStarted","Data":"a3d49d4d23ac30fddc1913199df3cdbf01a27ac0065c193ffa3d9b695456da3c"} Oct 30 01:45:43.844221 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.844184 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh"] Oct 30 01:45:43.891609 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.890038 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf"] Oct 30 01:45:43.891609 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.890374 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:45:43.891609 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.890455 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:45:43.891609 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.890501 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:45:43.894072 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.894032 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:45:43.894175 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.894128 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:45:43.894237 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:43.894194 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:45:44.288465 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:44.288434 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj" Oct 30 01:45:44.290915 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:44.290886 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj" Oct 30 01:45:44.291773 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:44.291744 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj" Oct 30 01:45:44.664739 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:44.664631 6221 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:45:44.664739 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:44.664709 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" podUID="48557095-c2dc-40cb-8ee3-8ee71567c61c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:45:44.746554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:44.745899 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf" event={"ID":"7b8071ca-a7f0-47a5-aabf-32148fa0463a","Type":"ContainerStarted","Data":"aabf3dfa503f01f3756aa6c952f83749c93e7e9e6c0152aa1f5c585cc5148fcc"} Oct 30 01:45:44.746554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:44.745990 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf" event={"ID":"7b8071ca-a7f0-47a5-aabf-32148fa0463a","Type":"ContainerStarted","Data":"eb0c7e1517808af08425d271f564d4a02517fc28da4c4a0e113b74ff10e5b09b"} Oct 30 01:45:44.748559 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:44.748082 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8f68ab-5bf445cc7d-9b2mq" event={"ID":"9abd0232-8d93-4dfa-ad61-7b01ed80eba8","Type":"ContainerStarted","Data":"c4a4a81ca87831dd912e63113ed33c6e2d116b11d805a6aee2905be69a936532"} Oct 30 01:45:44.748559 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:44.748351 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8f68ab-5bf445cc7d-9b2mq" Oct 30 01:45:44.750816 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:44.750792 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh" event={"ID":"59eaf4dd-3059-4fa3-bac8-93fc2b4d518a","Type":"ContainerStarted","Data":"7ed4f067ebd5c37303404d57b1d652dd61a80ce5364d9e8b2907dc535a0f954d"} Oct 30 01:45:44.750934 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:44.750839 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh" event={"ID":"59eaf4dd-3059-4fa3-bac8-93fc2b4d518a","Type":"ContainerStarted","Data":"34c7b2130b9c444193ed7a03ea256a0b3601de4bc56c2b2b2f418e5731b2209b"} Oct 30 01:45:44.795557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:44.793858 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf" podStartSLOduration=1.793839264 podStartE2EDuration="1.793839264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:44.792870837 +0000 UTC m=+1668.371158174" watchObservedRunningTime="2026-10-30 01:45:44.793839264 +0000 UTC m=+1668.372126598" Oct 30 01:45:44.898555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:44.896696 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh" podStartSLOduration=1.896675547 podStartE2EDuration="1.896675547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:44.893759156 +0000 UTC m=+1668.472046493" watchObservedRunningTime="2026-10-30 01:45:44.896675547 +0000 UTC m=+1668.474962884" Oct 30 01:45:44.898555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:44.898034 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8f68ab-5bf445cc7d-9b2mq" podStartSLOduration=2.574843806 podStartE2EDuration="2.89802113s" totalImagesPullingTime="323.177324ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:44.849000383 +0000 UTC m=+1668.427287720" watchObservedRunningTime="2026-10-30 01:45:44.89802113 +0000 UTC m=+1668.476308513" Oct 30 01:45:45.522555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:45.519895 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh" Oct 30 01:45:45.522555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:45.520168 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf" Oct 30 01:45:45.536059 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:45.534976 6221 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 01:45:45.536059 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:45.535034 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf" podUID="7b8071ca-a7f0-47a5-aabf-32148fa0463a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 01:45:45.536059 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:45.535095 6221 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 01:45:45.536059 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:45.535143 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh" podUID="59eaf4dd-3059-4fa3-bac8-93fc2b4d518a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 01:45:45.730681 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:45.730639 6221 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:45:45.739556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:45.730948 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" podUID="48557095-c2dc-40cb-8ee3-8ee71567c61c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:45:46.215815 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:46.215760 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" podUID="ea9583aa-8259-4c9b-932e-2d6559008d86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:45:46.267623 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:46.265507 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" Oct 30 01:45:46.272344 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:46.269635 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" Oct 30 01:45:46.272959 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:46.272844 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" Oct 30 01:45:46.363693 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:46.363647 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" podUID="018c3fe2-3e67-45ed-a3d7-d70f3fe5b956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:45:46.504025 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:46.503928 6221 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 01:45:46.504323 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:46.504293 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh" podUID="59eaf4dd-3059-4fa3-bac8-93fc2b4d518a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 01:45:46.525594 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:46.520930 6221 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 01:45:46.525594 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:46.521003 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf" podUID="7b8071ca-a7f0-47a5-aabf-32148fa0463a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 01:45:47.499509 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:47.499466 6221 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 01:45:47.499970 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:47.499523 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh" podUID="59eaf4dd-3059-4fa3-bac8-93fc2b4d518a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 01:45:47.515555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:47.514900 6221 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 01:45:47.515555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:47.514998 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf" podUID="7b8071ca-a7f0-47a5-aabf-32148fa0463a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 01:45:48.498881 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:48.498845 6221 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 01:45:48.499073 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:48.498912 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh" podUID="59eaf4dd-3059-4fa3-bac8-93fc2b4d518a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 01:45:48.538555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:48.537685 6221 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 01:45:48.538555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:48.537763 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf" podUID="7b8071ca-a7f0-47a5-aabf-32148fa0463a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 01:45:49.499447 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:49.499407 6221 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 01:45:49.499663 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:49.499461 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh" podUID="59eaf4dd-3059-4fa3-bac8-93fc2b4d518a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 01:45:49.535243 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:49.535202 6221 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 01:45:49.535510 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:49.535484 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf" podUID="7b8071ca-a7f0-47a5-aabf-32148fa0463a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 01:45:50.403922 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:50.403888 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n"] Oct 30 01:45:50.408088 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:50.408061 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n" Oct 30 01:45:50.441088 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:50.441050 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n"] Oct 30 01:45:50.499025 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:50.498992 6221 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 01:45:50.499258 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:50.499221 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh" podUID="59eaf4dd-3059-4fa3-bac8-93fc2b4d518a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 01:45:50.535057 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:50.534962 6221 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 01:45:50.535365 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:50.535329 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf" podUID="7b8071ca-a7f0-47a5-aabf-32148fa0463a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 01:45:50.545487 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:50.545457 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39e768cb-a319-4661-8e6d-aa7d654bba94-istio-data\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n\" (UID: \"39e768cb-a319-4661-8e6d-aa7d654bba94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n" Oct 30 01:45:50.545723 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:50.545699 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39e768cb-a319-4661-8e6d-aa7d654bba94-workload-certs\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n\" (UID: \"39e768cb-a319-4661-8e6d-aa7d654bba94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n" Oct 30 01:45:50.545816 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:50.545773 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39e768cb-a319-4661-8e6d-aa7d654bba94-istiod-ca-cert\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n\" (UID: \"39e768cb-a319-4661-8e6d-aa7d654bba94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n" Oct 30 01:45:50.545816 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:50.545803 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l9qt\" (UniqueName: \"kubernetes.io/projected/39e768cb-a319-4661-8e6d-aa7d654bba94-kube-api-access-8l9qt\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n\" (UID: \"39e768cb-a319-4661-8e6d-aa7d654bba94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n" Oct 30 01:45:50.545929 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:50.545834 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39e768cb-a319-4661-8e6d-aa7d654bba94-istio-envoy\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n\" (UID: \"39e768cb-a319-4661-8e6d-aa7d654bba94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n" Oct 30 01:45:50.545929 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:50.545867 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39e768cb-a319-4661-8e6d-aa7d654bba94-workload-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n\" (UID: \"39e768cb-a319-4661-8e6d-aa7d654bba94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n" Oct 30 01:45:50.546028 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:50.545927 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39e768cb-a319-4661-8e6d-aa7d654bba94-istio-podinfo\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n\" (UID: \"39e768cb-a319-4661-8e6d-aa7d654bba94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n" Oct 30 01:45:50.546077 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:50.546029 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39e768cb-a319-4661-8e6d-aa7d654bba94-credential-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n\" (UID: \"39e768cb-a319-4661-8e6d-aa7d654bba94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n" Oct 30 01:45:50.546125 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:50.546108 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39e768cb-a319-4661-8e6d-aa7d654bba94-istio-token\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n\" (UID: \"39e768cb-a319-4661-8e6d-aa7d654bba94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n" Oct 30 01:45:50.647160 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:50.647105 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39e768cb-a319-4661-8e6d-aa7d654bba94-credential-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n\" (UID: \"39e768cb-a319-4661-8e6d-aa7d654bba94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n" Oct 30 01:45:50.647373 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:50.647204 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39e768cb-a319-4661-8e6d-aa7d654bba94-istio-token\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n\" (UID: \"39e768cb-a319-4661-8e6d-aa7d654bba94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n" Oct 30 01:45:50.647373 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:50.647270 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39e768cb-a319-4661-8e6d-aa7d654bba94-istio-data\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n\" (UID: \"39e768cb-a319-4661-8e6d-aa7d654bba94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n" Oct 30 01:45:50.647505 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:50.647412 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39e768cb-a319-4661-8e6d-aa7d654bba94-workload-certs\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n\" (UID: \"39e768cb-a319-4661-8e6d-aa7d654bba94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n" Oct 30 01:45:50.647505 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:50.647454 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39e768cb-a319-4661-8e6d-aa7d654bba94-istiod-ca-cert\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n\" (UID: \"39e768cb-a319-4661-8e6d-aa7d654bba94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n" Oct 30 01:45:50.647505 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:50.647481 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l9qt\" (UniqueName: \"kubernetes.io/projected/39e768cb-a319-4661-8e6d-aa7d654bba94-kube-api-access-8l9qt\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n\" (UID: \"39e768cb-a319-4661-8e6d-aa7d654bba94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n" Oct 30 01:45:50.647746 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:50.647512 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39e768cb-a319-4661-8e6d-aa7d654bba94-istio-envoy\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n\" (UID: \"39e768cb-a319-4661-8e6d-aa7d654bba94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n" Oct 30 01:45:50.647746 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:50.647562 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39e768cb-a319-4661-8e6d-aa7d654bba94-workload-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n\" (UID: \"39e768cb-a319-4661-8e6d-aa7d654bba94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n" Oct 30 01:45:50.647746 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:50.647571 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39e768cb-a319-4661-8e6d-aa7d654bba94-credential-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n\" (UID: \"39e768cb-a319-4661-8e6d-aa7d654bba94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n" Oct 30 01:45:50.647746 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:50.647682 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39e768cb-a319-4661-8e6d-aa7d654bba94-istio-data\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n\" (UID: \"39e768cb-a319-4661-8e6d-aa7d654bba94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n" Oct 30 01:45:50.647976 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:50.647961 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39e768cb-a319-4661-8e6d-aa7d654bba94-workload-socket\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n\" (UID: \"39e768cb-a319-4661-8e6d-aa7d654bba94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n" Oct 30 01:45:50.648141 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:50.648117 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39e768cb-a319-4661-8e6d-aa7d654bba94-istio-podinfo\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n\" (UID: \"39e768cb-a319-4661-8e6d-aa7d654bba94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n" Oct 30 01:45:50.648258 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:50.648233 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39e768cb-a319-4661-8e6d-aa7d654bba94-istiod-ca-cert\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n\" (UID: \"39e768cb-a319-4661-8e6d-aa7d654bba94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n" Oct 30 01:45:50.648322 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:50.648272 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39e768cb-a319-4661-8e6d-aa7d654bba94-workload-certs\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n\" (UID: \"39e768cb-a319-4661-8e6d-aa7d654bba94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n" Oct 30 01:45:50.650287 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:50.650258 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39e768cb-a319-4661-8e6d-aa7d654bba94-istio-envoy\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n\" (UID: \"39e768cb-a319-4661-8e6d-aa7d654bba94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n" Oct 30 01:45:50.650888 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:50.650866 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39e768cb-a319-4661-8e6d-aa7d654bba94-istio-podinfo\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n\" (UID: \"39e768cb-a319-4661-8e6d-aa7d654bba94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n" Oct 30 01:45:50.666986 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:50.666908 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39e768cb-a319-4661-8e6d-aa7d654bba94-istio-token\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n\" (UID: \"39e768cb-a319-4661-8e6d-aa7d654bba94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n" Oct 30 01:45:50.667602 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:50.667579 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l9qt\" (UniqueName: \"kubernetes.io/projected/39e768cb-a319-4661-8e6d-aa7d654bba94-kube-api-access-8l9qt\") pod \"e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n\" (UID: \"39e768cb-a319-4661-8e6d-aa7d654bba94\") " pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n" Oct 30 01:45:50.798136 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:50.798090 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n" Oct 30 01:45:50.835771 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:50.835743 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8f68ab-5bf445cc7d-9b2mq" Oct 30 01:45:51.134084 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:51.133948 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n"] Oct 30 01:45:51.137976 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:45:51.137938 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39e768cb_a319_4661_8e6d_aa7d654bba94.slice/crio-ce0abdf73f1fc6f88036ffb2ea80ddbafd4de1c3afccb1ff8bb38d265ff2476a WatchSource:0}: Error finding container ce0abdf73f1fc6f88036ffb2ea80ddbafd4de1c3afccb1ff8bb38d265ff2476a: Status 404 returned error can't find the container with id ce0abdf73f1fc6f88036ffb2ea80ddbafd4de1c3afccb1ff8bb38d265ff2476a Oct 30 01:45:51.142276 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:51.142240 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:45:51.142371 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:51.142324 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:45:51.142371 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:51.142366 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:45:51.498864 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:45:51.498771 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:45:51.499545 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:51.499508 6221 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body= Oct 30 01:45:51.499731 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:51.499711 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh" podUID="59eaf4dd-3059-4fa3-bac8-93fc2b4d518a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" Oct 30 01:45:51.512384 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:51.512349 6221 patch_prober.go:29] interesting pod/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 30 01:45:51.512509 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:51.512414 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf" podUID="7b8071ca-a7f0-47a5-aabf-32148fa0463a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 30 01:45:51.942556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:51.941752 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n" event={"ID":"39e768cb-a319-4661-8e6d-aa7d654bba94","Type":"ContainerStarted","Data":"fa6e37df39057bb97547c3f2ee79ac08ee8c7052a9c8d016c749718149cac5d3"} Oct 30 01:45:51.942556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:51.941825 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n" event={"ID":"39e768cb-a319-4661-8e6d-aa7d654bba94","Type":"ContainerStarted","Data":"ce0abdf73f1fc6f88036ffb2ea80ddbafd4de1c3afccb1ff8bb38d265ff2476a"} Oct 30 01:45:52.025554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:52.025468 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n" podStartSLOduration=2.025446007 podStartE2EDuration="2.025446007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:45:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:45:52.005832542 +0000 UTC m=+1675.584119879" watchObservedRunningTime="2026-10-30 01:45:52.025446007 +0000 UTC m=+1675.603733344" Oct 30 01:45:52.503365 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:52.503329 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh" Oct 30 01:45:52.503755 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:52.503652 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh" Oct 30 01:45:52.504259 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:52.504239 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh" Oct 30 01:45:52.536088 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:52.536055 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf" Oct 30 01:45:52.536364 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:52.536344 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf" Oct 30 01:45:52.537128 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:52.537114 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf" Oct 30 01:45:52.804828 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:52.801199 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n" Oct 30 01:45:52.816556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:52.807658 6221 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 01:45:52.816556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:52.807723 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n" podUID="39e768cb-a319-4661-8e6d-aa7d654bba94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 01:45:53.799013 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:53.798976 6221 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 01:45:53.799435 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:53.799028 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n" podUID="39e768cb-a319-4661-8e6d-aa7d654bba94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 01:45:54.799148 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:54.799105 6221 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 01:45:54.799589 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:54.799163 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n" podUID="39e768cb-a319-4661-8e6d-aa7d654bba94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 01:45:55.798881 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:55.798841 6221 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 01:45:55.799037 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:55.798902 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n" podUID="39e768cb-a319-4661-8e6d-aa7d654bba94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 01:45:55.959513 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:55.959477 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0ae258-56f894cd47-vmwzz"] Oct 30 01:45:56.799202 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:56.799169 6221 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 01:45:56.799394 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:56.799221 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n" podUID="39e768cb-a319-4661-8e6d-aa7d654bba94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 01:45:57.799069 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:57.799032 6221 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 01:45:57.799463 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:57.799100 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n" podUID="39e768cb-a319-4661-8e6d-aa7d654bba94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 01:45:58.803038 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:58.803005 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n" Oct 30 01:45:58.803525 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:58.803410 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n" Oct 30 01:45:58.804116 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:45:58.804094 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n" Oct 30 01:46:00.966561 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:00.963257 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0ae258-56f894cd47-vmwzz" podUID="6fa64c39-aa67-4869-9164-a3edee1c0be8" containerName="payload-pre-processing" containerID="cri-o://5ad1a35d6242c4f3006835ed52c303d74ef58455486434c6f51770e9af44f6eb" gracePeriod=25 Oct 30 01:46:01.181556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:01.180664 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" podUID="ea9583aa-8259-4c9b-932e-2d6559008d86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:01.235744 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:01.235639 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0ae258-56f894cd47-vmwzz" podUID="6fa64c39-aa67-4869-9164-a3edee1c0be8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:46:01.283677 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:01.283624 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" podUID="354ec282-b972-4fca-b4e3-1e33ef461f2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:01.299685 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:01.299620 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" podUID="018c3fe2-3e67-45ed-a3d7-d70f3fe5b956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:01.615699 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:01.615650 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-0ae258/praxis-route-0ae258-kserve-fb65bffff-s8cc2" podUID="532d4bd3-5ece-42da-876f-496110860d4d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.127:8000/ready\": context deadline exceeded" Oct 30 01:46:01.690745 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:01.690682 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk" podUID="f4ae494e-81ec-4039-ae07-7ed1bf345f0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:01.750849 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:01.750791 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" podUID="9df52a01-0c8d-44ba-9fd7-9cb2fa41947a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:01.964749 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:01.964645 6221 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-praxis-a-0ae258/praxis-route-0ae258-kserve-fb65bffff-s8cc2" podUID="532d4bd3-5ece-42da-876f-496110860d4d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.127:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:02.140556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:02.132909 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" podUID="3b3c0e08-1371-4184-816a-d8b501526190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:02.183879 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:02.183807 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" podUID="ea9583aa-8259-4c9b-932e-2d6559008d86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:02.205225 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:02.205188 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" Oct 30 01:46:02.340557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:02.313690 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" podUID="354ec282-b972-4fca-b4e3-1e33ef461f2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:02.340557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:02.314882 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" Oct 30 01:46:02.340557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:02.320663 6221 patch_prober.go:29] interesting pod/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:46:02.340557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:02.320758 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" podUID="48557095-c2dc-40cb-8ee3-8ee71567c61c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:02.350558 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:02.342639 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" podUID="018c3fe2-3e67-45ed-a3d7-d70f3fe5b956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:02.368559 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:02.359756 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" Oct 30 01:46:02.642635 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:02.642412 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-0ae258/praxis-route-0ae258-kserve-fb65bffff-s8cc2" podUID="532d4bd3-5ece-42da-876f-496110860d4d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.127:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:02.720554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:02.695866 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk" podUID="f4ae494e-81ec-4039-ae07-7ed1bf345f0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:02.775606 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:02.755996 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" podUID="9df52a01-0c8d-44ba-9fd7-9cb2fa41947a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:02.875802 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:02.875748 6221 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" podUID="3b3c0e08-1371-4184-816a-d8b501526190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:02.956260 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:02.956157 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-0ae258/praxis-route-0ae258-kserve-fb65bffff-s8cc2" Oct 30 01:46:02.960605 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:02.956518 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk" Oct 30 01:46:02.960886 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:02.960868 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" Oct 30 01:46:03.015039 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:03.014983 6221 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" podUID="354ec282-b972-4fca-b4e3-1e33ef461f2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:03.156590 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:03.155851 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" podUID="3b3c0e08-1371-4184-816a-d8b501526190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:03.185736 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:03.174220 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" Oct 30 01:46:03.211706 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:03.205494 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" podUID="ea9583aa-8259-4c9b-932e-2d6559008d86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 01:46:03.318639 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:03.318577 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" podUID="354ec282-b972-4fca-b4e3-1e33ef461f2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:03.372743 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:03.372694 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" podUID="018c3fe2-3e67-45ed-a3d7-d70f3fe5b956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 01:46:03.668601 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:03.666683 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-0ae258/praxis-route-0ae258-kserve-fb65bffff-s8cc2" podUID="532d4bd3-5ece-42da-876f-496110860d4d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.127:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:03.703551 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:03.698630 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk" podUID="f4ae494e-81ec-4039-ae07-7ed1bf345f0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:03.853680 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:03.853619 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" podUID="9df52a01-0c8d-44ba-9fd7-9cb2fa41947a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:04.156698 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:04.156644 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" podUID="3b3c0e08-1371-4184-816a-d8b501526190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:04.212483 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:04.212346 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" podUID="ea9583aa-8259-4c9b-932e-2d6559008d86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 01:46:04.289297 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:04.289246 6221 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" podUID="ea9583aa-8259-4c9b-932e-2d6559008d86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:04.337572 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:04.337241 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" podUID="354ec282-b972-4fca-b4e3-1e33ef461f2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 01:46:04.382254 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:04.379304 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" podUID="018c3fe2-3e67-45ed-a3d7-d70f3fe5b956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:04.431627 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:04.429449 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk" Oct 30 01:46:04.439748 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:04.438933 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" Oct 30 01:46:04.449229 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:04.449206 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" Oct 30 01:46:04.455572 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:04.452806 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" Oct 30 01:46:04.478626 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:04.461716 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" Oct 30 01:46:04.478626 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:04.478143 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-0ae258/praxis-route-0ae258-kserve-fb65bffff-s8cc2" Oct 30 01:46:04.489626 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:04.489598 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" Oct 30 01:46:11.197883 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:11.197836 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0ae258-56f894cd47-vmwzz" podUID="6fa64c39-aa67-4869-9164-a3edee1c0be8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 01:46:12.189660 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:12.189624 6221 patch_prober.go:29] interesting pod/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:46:12.191874 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:12.189697 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" podUID="cbbfc7ea-d855-4b77-b484-b0b316f40963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:12.505203 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:12.505099 6221 generic.go:356] "Generic (PLEG): container finished" podID="cbbfc7ea-d855-4b77-b484-b0b316f40963" containerID="443dc49ae30b6ac2faf2336d59c3ed039cae65ec6c66df5f05ed79e88a85a8cc" exitCode=0 Oct 30 01:46:12.505203 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:12.505182 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" event={"ID":"cbbfc7ea-d855-4b77-b484-b0b316f40963","Type":"ContainerDied","Data":"443dc49ae30b6ac2faf2336d59c3ed039cae65ec6c66df5f05ed79e88a85a8cc"} Oct 30 01:46:12.505785 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:12.505766 6221 scope.go:122] "RemoveContainer" containerID="443dc49ae30b6ac2faf2336d59c3ed039cae65ec6c66df5f05ed79e88a85a8cc" Oct 30 01:46:12.507671 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:12.507637 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:46:12.507790 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:12.507725 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:46:12.507790 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:12.507770 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:46:13.547331 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:13.547294 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" event={"ID":"cbbfc7ea-d855-4b77-b484-b0b316f40963","Type":"ContainerStarted","Data":"09281d86ec47149a0eef45e57814712b7d5d7dbc05a8e778b588778dd32984f1"} Oct 30 01:46:14.371955 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:14.371917 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" Oct 30 01:46:14.372146 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:14.372075 6221 patch_prober.go:29] interesting pod/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 01:46:14.372146 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:14.372120 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" podUID="cbbfc7ea-d855-4b77-b484-b0b316f40963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 01:46:14.935079 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:14.935042 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:46:15.372289 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:15.372254 6221 patch_prober.go:29] interesting pod/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 01:46:15.372476 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:15.372316 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" podUID="cbbfc7ea-d855-4b77-b484-b0b316f40963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 01:46:16.376063 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:16.376024 6221 patch_prober.go:29] interesting pod/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 01:46:16.376741 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:16.376695 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" podUID="cbbfc7ea-d855-4b77-b484-b0b316f40963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 01:46:17.256764 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:17.256712 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" podUID="ea9583aa-8259-4c9b-932e-2d6559008d86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:17.342670 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:17.342621 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" podUID="354ec282-b972-4fca-b4e3-1e33ef461f2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:17.372979 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:17.372941 6221 patch_prober.go:29] interesting pod/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 01:46:17.374295 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:17.374262 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" podUID="cbbfc7ea-d855-4b77-b484-b0b316f40963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 01:46:17.374873 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:17.374845 6221 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk" podUID="f4ae494e-81ec-4039-ae07-7ed1bf345f0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:17.441621 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:17.441306 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" podUID="3b3c0e08-1371-4184-816a-d8b501526190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:17.474865 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:17.474817 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-0ae258/praxis-route-0ae258-kserve-fb65bffff-s8cc2" podUID="532d4bd3-5ece-42da-876f-496110860d4d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.127:8000/ready\": context deadline exceeded" Oct 30 01:46:17.502416 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:17.502358 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" podUID="018c3fe2-3e67-45ed-a3d7-d70f3fe5b956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 01:46:17.710695 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:17.710638 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk" podUID="f4ae494e-81ec-4039-ae07-7ed1bf345f0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:17.904555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:17.897974 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" podUID="9df52a01-0c8d-44ba-9fd7-9cb2fa41947a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 01:46:18.284561 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:18.282875 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" podUID="ea9583aa-8259-4c9b-932e-2d6559008d86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 01:46:18.284561 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:18.283610 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" Oct 30 01:46:18.369555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:18.347270 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" podUID="354ec282-b972-4fca-b4e3-1e33ef461f2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 01:46:18.369555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:18.353306 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" Oct 30 01:46:18.379330 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:18.379294 6221 patch_prober.go:29] interesting pod/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 01:46:18.379583 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:18.379558 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" podUID="cbbfc7ea-d855-4b77-b484-b0b316f40963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 01:46:18.514740 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:18.514684 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" podUID="018c3fe2-3e67-45ed-a3d7-d70f3fe5b956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:18.715727 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:18.715627 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk" podUID="f4ae494e-81ec-4039-ae07-7ed1bf345f0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:18.809584 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:18.800839 6221 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" podUID="018c3fe2-3e67-45ed-a3d7-d70f3fe5b956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 01:46:18.930557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:18.912290 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" Oct 30 01:46:18.930956 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:18.917883 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" podUID="9df52a01-0c8d-44ba-9fd7-9cb2fa41947a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 30 01:46:18.934499 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:18.934478 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk" Oct 30 01:46:18.935197 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:18.935180 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" Oct 30 01:46:19.333557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:19.307572 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" podUID="ea9583aa-8259-4c9b-932e-2d6559008d86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 01:46:19.363042 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:19.362999 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" podUID="354ec282-b972-4fca-b4e3-1e33ef461f2f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 30 01:46:19.458199 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:19.458166 6221 patch_prober.go:29] interesting pod/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 01:46:19.458460 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:19.458436 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" podUID="cbbfc7ea-d855-4b77-b484-b0b316f40963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 01:46:19.567322 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:19.548056 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x"] Oct 30 01:46:19.613638 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:19.611860 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x" Oct 30 01:46:19.668555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:19.645624 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" podUID="018c3fe2-3e67-45ed-a3d7-d70f3fe5b956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 01:46:19.690652 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:19.687388 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" Oct 30 01:46:19.730556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:19.716897 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk" podUID="f4ae494e-81ec-4039-ae07-7ed1bf345f0f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Oct 30 01:46:19.730556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:19.717761 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x"] Oct 30 01:46:19.761951 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:19.761743 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0f654ff-538a-41ae-ab41-f3e8d658949d-workload-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x\" (UID: \"e0f654ff-538a-41ae-ab41-f3e8d658949d\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x" Oct 30 01:46:19.762297 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:19.762271 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9vtq\" (UniqueName: \"kubernetes.io/projected/e0f654ff-538a-41ae-ab41-f3e8d658949d-kube-api-access-m9vtq\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x\" (UID: \"e0f654ff-538a-41ae-ab41-f3e8d658949d\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x" Oct 30 01:46:19.762458 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:19.762441 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0f654ff-538a-41ae-ab41-f3e8d658949d-istio-data\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x\" (UID: \"e0f654ff-538a-41ae-ab41-f3e8d658949d\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x" Oct 30 01:46:19.762660 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:19.762637 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0f654ff-538a-41ae-ab41-f3e8d658949d-workload-certs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x\" (UID: \"e0f654ff-538a-41ae-ab41-f3e8d658949d\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x" Oct 30 01:46:19.762828 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:19.762811 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0f654ff-538a-41ae-ab41-f3e8d658949d-istio-envoy\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x\" (UID: \"e0f654ff-538a-41ae-ab41-f3e8d658949d\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x" Oct 30 01:46:19.762967 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:19.762930 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0f654ff-538a-41ae-ab41-f3e8d658949d-istio-podinfo\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x\" (UID: \"e0f654ff-538a-41ae-ab41-f3e8d658949d\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x" Oct 30 01:46:19.763129 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:19.763113 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0f654ff-538a-41ae-ab41-f3e8d658949d-istiod-ca-cert\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x\" (UID: \"e0f654ff-538a-41ae-ab41-f3e8d658949d\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x" Oct 30 01:46:19.763294 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:19.763275 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0f654ff-538a-41ae-ab41-f3e8d658949d-credential-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x\" (UID: \"e0f654ff-538a-41ae-ab41-f3e8d658949d\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x" Oct 30 01:46:19.763447 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:19.763429 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0f654ff-538a-41ae-ab41-f3e8d658949d-istio-token\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x\" (UID: \"e0f654ff-538a-41ae-ab41-f3e8d658949d\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x" Oct 30 01:46:19.813157 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:19.813128 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" Oct 30 01:46:19.874087 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:19.872908 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0f654ff-538a-41ae-ab41-f3e8d658949d-istiod-ca-cert\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x\" (UID: \"e0f654ff-538a-41ae-ab41-f3e8d658949d\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x" Oct 30 01:46:19.874087 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:19.872993 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0f654ff-538a-41ae-ab41-f3e8d658949d-credential-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x\" (UID: \"e0f654ff-538a-41ae-ab41-f3e8d658949d\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x" Oct 30 01:46:19.874087 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:19.873052 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0f654ff-538a-41ae-ab41-f3e8d658949d-istio-token\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x\" (UID: \"e0f654ff-538a-41ae-ab41-f3e8d658949d\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x" Oct 30 01:46:19.874087 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:19.873143 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0f654ff-538a-41ae-ab41-f3e8d658949d-workload-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x\" (UID: \"e0f654ff-538a-41ae-ab41-f3e8d658949d\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x" Oct 30 01:46:19.874087 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:19.873217 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9vtq\" (UniqueName: \"kubernetes.io/projected/e0f654ff-538a-41ae-ab41-f3e8d658949d-kube-api-access-m9vtq\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x\" (UID: \"e0f654ff-538a-41ae-ab41-f3e8d658949d\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x" Oct 30 01:46:19.874087 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:19.873247 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0f654ff-538a-41ae-ab41-f3e8d658949d-istio-data\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x\" (UID: \"e0f654ff-538a-41ae-ab41-f3e8d658949d\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x" Oct 30 01:46:19.874087 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:19.873324 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0f654ff-538a-41ae-ab41-f3e8d658949d-workload-certs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x\" (UID: \"e0f654ff-538a-41ae-ab41-f3e8d658949d\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x" Oct 30 01:46:19.874087 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:19.873397 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0f654ff-538a-41ae-ab41-f3e8d658949d-istio-envoy\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x\" (UID: \"e0f654ff-538a-41ae-ab41-f3e8d658949d\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x" Oct 30 01:46:19.874087 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:19.873426 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0f654ff-538a-41ae-ab41-f3e8d658949d-istio-podinfo\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x\" (UID: \"e0f654ff-538a-41ae-ab41-f3e8d658949d\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x" Oct 30 01:46:19.882023 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:19.881991 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0f654ff-538a-41ae-ab41-f3e8d658949d-workload-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x\" (UID: \"e0f654ff-538a-41ae-ab41-f3e8d658949d\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x" Oct 30 01:46:19.882944 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:19.882916 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0f654ff-538a-41ae-ab41-f3e8d658949d-istiod-ca-cert\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x\" (UID: \"e0f654ff-538a-41ae-ab41-f3e8d658949d\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x" Oct 30 01:46:19.883366 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:19.883345 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0f654ff-538a-41ae-ab41-f3e8d658949d-credential-socket\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x\" (UID: \"e0f654ff-538a-41ae-ab41-f3e8d658949d\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x" Oct 30 01:46:19.884052 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:19.884029 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0f654ff-538a-41ae-ab41-f3e8d658949d-istio-data\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x\" (UID: \"e0f654ff-538a-41ae-ab41-f3e8d658949d\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x" Oct 30 01:46:19.884676 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:19.884656 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0f654ff-538a-41ae-ab41-f3e8d658949d-workload-certs\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x\" (UID: \"e0f654ff-538a-41ae-ab41-f3e8d658949d\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x" Oct 30 01:46:19.985838 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:19.985808 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" Oct 30 01:46:20.059333 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:20.058757 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0f654ff-538a-41ae-ab41-f3e8d658949d-istio-envoy\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x\" (UID: \"e0f654ff-538a-41ae-ab41-f3e8d658949d\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x" Oct 30 01:46:20.059940 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:20.059913 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0f654ff-538a-41ae-ab41-f3e8d658949d-istio-podinfo\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x\" (UID: \"e0f654ff-538a-41ae-ab41-f3e8d658949d\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x" Oct 30 01:46:20.063602 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:20.062070 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0f654ff-538a-41ae-ab41-f3e8d658949d-istio-token\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x\" (UID: \"e0f654ff-538a-41ae-ab41-f3e8d658949d\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x" Oct 30 01:46:20.063602 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:20.060466 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9vtq\" (UniqueName: \"kubernetes.io/projected/e0f654ff-538a-41ae-ab41-f3e8d658949d-kube-api-access-m9vtq\") pod \"e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x\" (UID: \"e0f654ff-538a-41ae-ab41-f3e8d658949d\") " pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x" Oct 30 01:46:20.065394 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:20.065364 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" Oct 30 01:46:20.072580 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:20.069617 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk" Oct 30 01:46:20.140607 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:20.139765 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x" Oct 30 01:46:20.244095 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:20.243940 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f197b1-794699547-t9qsc"] Oct 30 01:46:20.372702 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:20.372670 6221 patch_prober.go:29] interesting pod/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 01:46:20.372873 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:20.372717 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" podUID="cbbfc7ea-d855-4b77-b484-b0b316f40963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 01:46:20.551164 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:46:20.551122 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0f654ff_538a_41ae_ab41_f3e8d658949d.slice/crio-42a2e1034d153cf2da116c9d64f3adad2ef1605f66f4b1840abaf7057b9fa5dc WatchSource:0}: Error finding container 42a2e1034d153cf2da116c9d64f3adad2ef1605f66f4b1840abaf7057b9fa5dc: Status 404 returned error can't find the container with id 42a2e1034d153cf2da116c9d64f3adad2ef1605f66f4b1840abaf7057b9fa5dc Oct 30 01:46:20.551854 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:20.551834 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x"] Oct 30 01:46:20.553960 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:20.553930 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:46:20.554044 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:20.553992 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:46:20.554044 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:20.554022 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:46:20.969609 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:20.966023 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x" event={"ID":"e0f654ff-538a-41ae-ab41-f3e8d658949d","Type":"ContainerStarted","Data":"86f18a15f86d84107977e6620573b962bc905c28fbcbbd5dd1c50818c2a8d6b9"} Oct 30 01:46:20.969609 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:20.966130 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x" event={"ID":"e0f654ff-538a-41ae-ab41-f3e8d658949d","Type":"ContainerStarted","Data":"42a2e1034d153cf2da116c9d64f3adad2ef1605f66f4b1840abaf7057b9fa5dc"} Oct 30 01:46:21.015051 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:21.014903 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x" podStartSLOduration=2.014860713 podStartE2EDuration="2.014860713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:46:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:46:21.012438111 +0000 UTC m=+1704.590725477" watchObservedRunningTime="2026-10-30 01:46:21.014860713 +0000 UTC m=+1704.593148051" Oct 30 01:46:21.141449 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:21.141403 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x" Oct 30 01:46:21.142730 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:21.142704 6221 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 01:46:21.142843 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:21.142764 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x" podUID="e0f654ff-538a-41ae-ab41-f3e8d658949d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 01:46:21.180109 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:21.180084 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0ae258-56f894cd47-vmwzz" Oct 30 01:46:21.213026 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:21.212994 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fa64c39-aa67-4869-9164-a3edee1c0be8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fa64c39-aa67-4869-9164-a3edee1c0be8-plugins-config-volume\") pod \"6fa64c39-aa67-4869-9164-a3edee1c0be8\" (UID: \"6fa64c39-aa67-4869-9164-a3edee1c0be8\") " Oct 30 01:46:21.213181 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:21.213085 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fa64c39-aa67-4869-9164-a3edee1c0be8-kube-api-access-x8bfq\" (UniqueName: \"kubernetes.io/projected/6fa64c39-aa67-4869-9164-a3edee1c0be8-kube-api-access-x8bfq\") pod \"6fa64c39-aa67-4869-9164-a3edee1c0be8\" (UID: \"6fa64c39-aa67-4869-9164-a3edee1c0be8\") " Oct 30 01:46:21.213345 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:21.213324 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fa64c39-aa67-4869-9164-a3edee1c0be8-plugins-config-volume" pod "6fa64c39-aa67-4869-9164-a3edee1c0be8" (UID: "6fa64c39-aa67-4869-9164-a3edee1c0be8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:46:21.213718 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:21.213699 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fa64c39-aa67-4869-9164-a3edee1c0be8-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:46:21.215396 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:21.215363 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fa64c39-aa67-4869-9164-a3edee1c0be8-kube-api-access-x8bfq" pod "6fa64c39-aa67-4869-9164-a3edee1c0be8" (UID: "6fa64c39-aa67-4869-9164-a3edee1c0be8"). InnerVolumeSpecName "kube-api-access-x8bfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:46:21.324561 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:21.322978 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8bfq\" (UniqueName: \"kubernetes.io/projected/6fa64c39-aa67-4869-9164-a3edee1c0be8-kube-api-access-x8bfq\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:46:21.395574 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:21.395390 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" Oct 30 01:46:21.399473 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:21.397025 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" Oct 30 01:46:21.399473 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:21.398640 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" Oct 30 01:46:21.602103 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:46:21.602022 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:46:21.974218 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:21.974129 6221 generic.go:356] "Generic (PLEG): container finished" podID="6fa64c39-aa67-4869-9164-a3edee1c0be8" containerID="5ad1a35d6242c4f3006835ed52c303d74ef58455486434c6f51770e9af44f6eb" exitCode=0 Oct 30 01:46:21.974218 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:21.974176 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0ae258-56f894cd47-vmwzz" Oct 30 01:46:21.974218 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:21.974208 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0ae258-56f894cd47-vmwzz" event={"ID":"6fa64c39-aa67-4869-9164-a3edee1c0be8","Type":"ContainerDied","Data":"5ad1a35d6242c4f3006835ed52c303d74ef58455486434c6f51770e9af44f6eb"} Oct 30 01:46:21.974893 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:21.974264 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-0ae258-56f894cd47-vmwzz" event={"ID":"6fa64c39-aa67-4869-9164-a3edee1c0be8","Type":"ContainerDied","Data":"6898db5b2223b918d171d9e19ab4e392282f25567034432497eec8168a0aa0be"} Oct 30 01:46:21.974893 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:21.974292 6221 scope.go:122] "RemoveContainer" containerID="5ad1a35d6242c4f3006835ed52c303d74ef58455486434c6f51770e9af44f6eb" Oct 30 01:46:21.984859 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:21.984841 6221 scope.go:122] "RemoveContainer" containerID="5ad1a35d6242c4f3006835ed52c303d74ef58455486434c6f51770e9af44f6eb" Oct 30 01:46:21.985119 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:46:21.985100 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ad1a35d6242c4f3006835ed52c303d74ef58455486434c6f51770e9af44f6eb\": container with ID starting with 5ad1a35d6242c4f3006835ed52c303d74ef58455486434c6f51770e9af44f6eb not found: ID does not exist" containerID="5ad1a35d6242c4f3006835ed52c303d74ef58455486434c6f51770e9af44f6eb" Oct 30 01:46:21.985181 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:21.985126 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ad1a35d6242c4f3006835ed52c303d74ef58455486434c6f51770e9af44f6eb"} err="failed to get container status \"5ad1a35d6242c4f3006835ed52c303d74ef58455486434c6f51770e9af44f6eb\": rpc error: code = NotFound desc = could not find container \"5ad1a35d6242c4f3006835ed52c303d74ef58455486434c6f51770e9af44f6eb\": container with ID starting with 5ad1a35d6242c4f3006835ed52c303d74ef58455486434c6f51770e9af44f6eb not found: ID does not exist" Oct 30 01:46:22.033849 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:22.033728 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0ae258-56f894cd47-vmwzz"] Oct 30 01:46:22.038698 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:22.038672 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-0ae258-56f894cd47-vmwzz"] Oct 30 01:46:22.141160 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:22.141126 6221 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 01:46:22.141337 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:22.141177 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x" podUID="e0f654ff-538a-41ae-ab41-f3e8d658949d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 01:46:22.547423 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:22.547388 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fa64c39-aa67-4869-9164-a3edee1c0be8" path="/var/lib/kubelet/pods/6fa64c39-aa67-4869-9164-a3edee1c0be8/volumes" Oct 30 01:46:23.141314 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:23.141282 6221 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 01:46:23.141848 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:23.141810 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x" podUID="e0f654ff-538a-41ae-ab41-f3e8d658949d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 01:46:24.141629 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:24.141590 6221 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 01:46:24.142016 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:24.141648 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x" podUID="e0f654ff-538a-41ae-ab41-f3e8d658949d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 01:46:25.141103 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:25.141064 6221 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 01:46:25.141299 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:25.141123 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x" podUID="e0f654ff-538a-41ae-ab41-f3e8d658949d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 01:46:25.244385 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:25.244342 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f197b1-794699547-t9qsc" podUID="4a217cae-637d-4f32-908b-b9ea322a01e7" containerName="payload-pre-processing" containerID="cri-o://fce6adce05f07aa7776798e0b82b7536e898d97c6b04ef601f6cc34515a11531" gracePeriod=25 Oct 30 01:46:25.568939 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:25.568906 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f197b1-794699547-t9qsc" Oct 30 01:46:25.664018 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:25.663985 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a217cae-637d-4f32-908b-b9ea322a01e7-kube-api-access-qkxtv\" (UniqueName: \"kubernetes.io/projected/4a217cae-637d-4f32-908b-b9ea322a01e7-kube-api-access-qkxtv\") pod \"4a217cae-637d-4f32-908b-b9ea322a01e7\" (UID: \"4a217cae-637d-4f32-908b-b9ea322a01e7\") " Oct 30 01:46:25.664206 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:25.664056 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a217cae-637d-4f32-908b-b9ea322a01e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a217cae-637d-4f32-908b-b9ea322a01e7-plugins-config-volume\") pod \"4a217cae-637d-4f32-908b-b9ea322a01e7\" (UID: \"4a217cae-637d-4f32-908b-b9ea322a01e7\") " Oct 30 01:46:25.664521 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:25.664492 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a217cae-637d-4f32-908b-b9ea322a01e7-plugins-config-volume" pod "4a217cae-637d-4f32-908b-b9ea322a01e7" (UID: "4a217cae-637d-4f32-908b-b9ea322a01e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:46:25.666150 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:25.666122 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a217cae-637d-4f32-908b-b9ea322a01e7-kube-api-access-qkxtv" pod "4a217cae-637d-4f32-908b-b9ea322a01e7" (UID: "4a217cae-637d-4f32-908b-b9ea322a01e7"). InnerVolumeSpecName "kube-api-access-qkxtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:46:25.774605 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:25.768897 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkxtv\" (UniqueName: \"kubernetes.io/projected/4a217cae-637d-4f32-908b-b9ea322a01e7-kube-api-access-qkxtv\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:46:25.774605 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:25.768936 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a217cae-637d-4f32-908b-b9ea322a01e7-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:46:26.004331 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:26.004295 6221 generic.go:356] "Generic (PLEG): container finished" podID="4a217cae-637d-4f32-908b-b9ea322a01e7" containerID="fce6adce05f07aa7776798e0b82b7536e898d97c6b04ef601f6cc34515a11531" exitCode=0 Oct 30 01:46:26.004485 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:26.004340 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f197b1-794699547-t9qsc" Oct 30 01:46:26.004485 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:26.004378 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f197b1-794699547-t9qsc" event={"ID":"4a217cae-637d-4f32-908b-b9ea322a01e7","Type":"ContainerDied","Data":"fce6adce05f07aa7776798e0b82b7536e898d97c6b04ef601f6cc34515a11531"} Oct 30 01:46:26.004485 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:26.004427 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f197b1-794699547-t9qsc" event={"ID":"4a217cae-637d-4f32-908b-b9ea322a01e7","Type":"ContainerDied","Data":"4e616faf15975d7c190108f8787088b14c7205c756d86d0994eb8332cb867894"} Oct 30 01:46:26.004485 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:26.004450 6221 scope.go:122] "RemoveContainer" containerID="fce6adce05f07aa7776798e0b82b7536e898d97c6b04ef601f6cc34515a11531" Oct 30 01:46:26.046596 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:26.046555 6221 scope.go:122] "RemoveContainer" containerID="fce6adce05f07aa7776798e0b82b7536e898d97c6b04ef601f6cc34515a11531" Oct 30 01:46:26.046977 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:46:26.046946 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fce6adce05f07aa7776798e0b82b7536e898d97c6b04ef601f6cc34515a11531\": container with ID starting with fce6adce05f07aa7776798e0b82b7536e898d97c6b04ef601f6cc34515a11531 not found: ID does not exist" containerID="fce6adce05f07aa7776798e0b82b7536e898d97c6b04ef601f6cc34515a11531" Oct 30 01:46:26.047088 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:26.046986 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fce6adce05f07aa7776798e0b82b7536e898d97c6b04ef601f6cc34515a11531"} err="failed to get container status \"fce6adce05f07aa7776798e0b82b7536e898d97c6b04ef601f6cc34515a11531\": rpc error: code = NotFound desc = could not find container \"fce6adce05f07aa7776798e0b82b7536e898d97c6b04ef601f6cc34515a11531\": container with ID starting with fce6adce05f07aa7776798e0b82b7536e898d97c6b04ef601f6cc34515a11531 not found: ID does not exist" Oct 30 01:46:26.085936 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:26.085904 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f197b1-794699547-t9qsc"] Oct 30 01:46:26.100304 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:26.100271 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f197b1-794699547-t9qsc"] Oct 30 01:46:26.140881 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:26.140846 6221 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 01:46:26.141062 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:26.140898 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x" podUID="e0f654ff-538a-41ae-ab41-f3e8d658949d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 01:46:26.549113 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:26.549080 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a217cae-637d-4f32-908b-b9ea322a01e7" path="/var/lib/kubelet/pods/4a217cae-637d-4f32-908b-b9ea322a01e7/volumes" Oct 30 01:46:27.141268 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:27.141229 6221 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 01:46:27.141432 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:27.141286 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x" podUID="e0f654ff-538a-41ae-ab41-f3e8d658949d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 01:46:27.421150 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:27.421054 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-0ae258/praxis-route-0ae258-kserve-fb65bffff-s8cc2"] Oct 30 01:46:28.496719 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:28.496684 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x" Oct 30 01:46:28.497206 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:28.497128 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x" Oct 30 01:46:28.497956 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:28.497935 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x" Oct 30 01:46:28.830386 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:28.830352 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-396f8e-88448749f-cwwq8"] Oct 30 01:46:28.830954 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:28.830938 6221 cpu_manager.go:414] "containerMap: removing container" podUID="4a217cae-637d-4f32-908b-b9ea322a01e7" containerName="payload-pre-processing" Oct 30 01:46:28.831020 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:28.830955 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a217cae-637d-4f32-908b-b9ea322a01e7" containerName="payload-pre-processing" Oct 30 01:46:28.831020 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:28.830976 6221 cpu_manager.go:414] "containerMap: removing container" podUID="6fa64c39-aa67-4869-9164-a3edee1c0be8" containerName="payload-pre-processing" Oct 30 01:46:28.831020 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:28.830982 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fa64c39-aa67-4869-9164-a3edee1c0be8" containerName="payload-pre-processing" Oct 30 01:46:28.831116 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:28.831086 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a217cae-637d-4f32-908b-b9ea322a01e7" containerName="payload-pre-processing" Oct 30 01:46:28.831116 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:28.831098 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fa64c39-aa67-4869-9164-a3edee1c0be8" containerName="payload-pre-processing" Oct 30 01:46:28.835685 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:28.835665 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-396f8e-88448749f-cwwq8" Oct 30 01:46:28.856686 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:28.856658 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-396f8e-dockercfg-j9mhm\"" Oct 30 01:46:28.856896 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:28.856783 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-396f8e\"" Oct 30 01:46:28.859895 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:28.859871 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-396f8e-88448749f-cwwq8"] Oct 30 01:46:28.964020 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:28.963984 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-396f8e-c744b6894-4f4lh"] Oct 30 01:46:28.970982 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:28.970953 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-396f8e-c744b6894-4f4lh"] Oct 30 01:46:28.971341 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:28.971275 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-396f8e-c744b6894-4f4lh" Oct 30 01:46:29.050552 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:29.046664 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2d4fdd6-a883-4f5c-90bd-6f36b799ffa0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-396f8e-88448749f-cwwq8\" (UID: \"b2d4fdd6-a883-4f5c-90bd-6f36b799ffa0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-396f8e-88448749f-cwwq8" Oct 30 01:46:29.050552 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:29.046868 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57mjn\" (UniqueName: \"kubernetes.io/projected/b2d4fdd6-a883-4f5c-90bd-6f36b799ffa0-kube-api-access-57mjn\") pod \"payload-pre-processing-e2e-shared-a-w0-396f8e-88448749f-cwwq8\" (UID: \"b2d4fdd6-a883-4f5c-90bd-6f36b799ffa0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-396f8e-88448749f-cwwq8" Oct 30 01:46:29.148158 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:29.148068 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdlnr\" (UniqueName: \"kubernetes.io/projected/30b6e290-ede2-4b9f-b65d-64d7e14355c5-kube-api-access-kdlnr\") pod \"payload-processing-e2e-shared-a-w0-396f8e-c744b6894-4f4lh\" (UID: \"30b6e290-ede2-4b9f-b65d-64d7e14355c5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-396f8e-c744b6894-4f4lh" Oct 30 01:46:29.148158 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:29.148146 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57mjn\" (UniqueName: \"kubernetes.io/projected/b2d4fdd6-a883-4f5c-90bd-6f36b799ffa0-kube-api-access-57mjn\") pod \"payload-pre-processing-e2e-shared-a-w0-396f8e-88448749f-cwwq8\" (UID: \"b2d4fdd6-a883-4f5c-90bd-6f36b799ffa0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-396f8e-88448749f-cwwq8" Oct 30 01:46:29.148379 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:29.148297 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30b6e290-ede2-4b9f-b65d-64d7e14355c5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-396f8e-c744b6894-4f4lh\" (UID: \"30b6e290-ede2-4b9f-b65d-64d7e14355c5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-396f8e-c744b6894-4f4lh" Oct 30 01:46:29.148421 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:29.148393 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2d4fdd6-a883-4f5c-90bd-6f36b799ffa0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-396f8e-88448749f-cwwq8\" (UID: \"b2d4fdd6-a883-4f5c-90bd-6f36b799ffa0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-396f8e-88448749f-cwwq8" Oct 30 01:46:29.149045 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:29.149026 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2d4fdd6-a883-4f5c-90bd-6f36b799ffa0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-396f8e-88448749f-cwwq8\" (UID: \"b2d4fdd6-a883-4f5c-90bd-6f36b799ffa0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-396f8e-88448749f-cwwq8" Oct 30 01:46:29.170406 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:29.170371 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57mjn\" (UniqueName: \"kubernetes.io/projected/b2d4fdd6-a883-4f5c-90bd-6f36b799ffa0-kube-api-access-57mjn\") pod \"payload-pre-processing-e2e-shared-a-w0-396f8e-88448749f-cwwq8\" (UID: \"b2d4fdd6-a883-4f5c-90bd-6f36b799ffa0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-396f8e-88448749f-cwwq8" Oct 30 01:46:29.249483 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:29.249453 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdlnr\" (UniqueName: \"kubernetes.io/projected/30b6e290-ede2-4b9f-b65d-64d7e14355c5-kube-api-access-kdlnr\") pod \"payload-processing-e2e-shared-a-w0-396f8e-c744b6894-4f4lh\" (UID: \"30b6e290-ede2-4b9f-b65d-64d7e14355c5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-396f8e-c744b6894-4f4lh" Oct 30 01:46:29.249690 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:29.249654 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30b6e290-ede2-4b9f-b65d-64d7e14355c5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-396f8e-c744b6894-4f4lh\" (UID: \"30b6e290-ede2-4b9f-b65d-64d7e14355c5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-396f8e-c744b6894-4f4lh" Oct 30 01:46:29.250381 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:29.250358 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30b6e290-ede2-4b9f-b65d-64d7e14355c5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-396f8e-c744b6894-4f4lh\" (UID: \"30b6e290-ede2-4b9f-b65d-64d7e14355c5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-396f8e-c744b6894-4f4lh" Oct 30 01:46:29.273297 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:29.273275 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdlnr\" (UniqueName: \"kubernetes.io/projected/30b6e290-ede2-4b9f-b65d-64d7e14355c5-kube-api-access-kdlnr\") pod \"payload-processing-e2e-shared-a-w0-396f8e-c744b6894-4f4lh\" (UID: \"30b6e290-ede2-4b9f-b65d-64d7e14355c5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-396f8e-c744b6894-4f4lh" Oct 30 01:46:29.408980 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:29.408896 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-396f8e-c744b6894-4f4lh" Oct 30 01:46:29.449235 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:29.449203 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-396f8e-88448749f-cwwq8" Oct 30 01:46:29.746369 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:29.746329 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-396f8e-c744b6894-4f4lh"] Oct 30 01:46:29.801237 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:29.801213 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-396f8e-88448749f-cwwq8"] Oct 30 01:46:29.804108 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:46:29.804080 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2d4fdd6_a883_4f5c_90bd_6f36b799ffa0.slice/crio-50b54b313503ad56e21689a869b8e8a592aa5df43eca7b0921de56a708624758 WatchSource:0}: Error finding container 50b54b313503ad56e21689a869b8e8a592aa5df43eca7b0921de56a708624758: Status 404 returned error can't find the container with id 50b54b313503ad56e21689a869b8e8a592aa5df43eca7b0921de56a708624758 Oct 30 01:46:30.115846 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:30.115810 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-396f8e-88448749f-cwwq8" event={"ID":"b2d4fdd6-a883-4f5c-90bd-6f36b799ffa0","Type":"ContainerStarted","Data":"45cafb0ca65d1e7fc8c0a6163644325373636108311345d2b64ccd12ce8ef91a"} Oct 30 01:46:30.116057 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:30.115871 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-396f8e-88448749f-cwwq8" Oct 30 01:46:30.116057 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:30.115896 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-396f8e-88448749f-cwwq8" event={"ID":"b2d4fdd6-a883-4f5c-90bd-6f36b799ffa0","Type":"ContainerStarted","Data":"50b54b313503ad56e21689a869b8e8a592aa5df43eca7b0921de56a708624758"} Oct 30 01:46:30.117348 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:30.117315 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-396f8e-c744b6894-4f4lh" event={"ID":"30b6e290-ede2-4b9f-b65d-64d7e14355c5","Type":"ContainerStarted","Data":"3e505be50663b3c77d307fb99d54c4f5c4f4de6db876ce3bb465029e03dcfd41"} Oct 30 01:46:30.117480 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:30.117363 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-396f8e-c744b6894-4f4lh" event={"ID":"30b6e290-ede2-4b9f-b65d-64d7e14355c5","Type":"ContainerStarted","Data":"6168b948a1927f2d7484a04d9108cf14a7277eff0dd81594698dd385fc75ef45"} Oct 30 01:46:30.117480 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:30.117392 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-396f8e-c744b6894-4f4lh" Oct 30 01:46:30.166334 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:30.166281 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-396f8e-88448749f-cwwq8" podStartSLOduration=2.166267523 podStartE2EDuration="2.166267523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:46:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:46:30.163109609 +0000 UTC m=+1713.741396945" watchObservedRunningTime="2026-10-30 01:46:30.166267523 +0000 UTC m=+1713.744554855" Oct 30 01:46:30.232038 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:30.231994 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-396f8e-c744b6894-4f4lh" podStartSLOduration=2.231980593 podStartE2EDuration="2.231980593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:46:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:46:30.226385986 +0000 UTC m=+1713.804673322" watchObservedRunningTime="2026-10-30 01:46:30.231980593 +0000 UTC m=+1713.810267929" Oct 30 01:46:31.237761 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:31.237274 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f197b1-5847f96fcd-4frnn"] Oct 30 01:46:31.237761 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:31.237559 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f197b1-5847f96fcd-4frnn" podUID="d44c10f7-7694-4fdd-86ce-669947932edb" containerName="maas-api" containerID="cri-o://14fc89219800f091e02610ce4ba6f579d858d093e4bf8f9364b8f42f7f5caebe" gracePeriod=30 Oct 30 01:46:31.615572 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:31.607266 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f197b1-5847f96fcd-4frnn" Oct 30 01:46:31.732626 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:31.730086 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d44c10f7-7694-4fdd-86ce-669947932edb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d44c10f7-7694-4fdd-86ce-669947932edb-maas-api-tls\") pod \"d44c10f7-7694-4fdd-86ce-669947932edb\" (UID: \"d44c10f7-7694-4fdd-86ce-669947932edb\") " Oct 30 01:46:31.732626 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:31.730158 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d44c10f7-7694-4fdd-86ce-669947932edb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d44c10f7-7694-4fdd-86ce-669947932edb-openshift-service-ca\") pod \"d44c10f7-7694-4fdd-86ce-669947932edb\" (UID: \"d44c10f7-7694-4fdd-86ce-669947932edb\") " Oct 30 01:46:31.732626 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:31.730281 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d44c10f7-7694-4fdd-86ce-669947932edb-kube-api-access-z2p6c\" (UniqueName: \"kubernetes.io/projected/d44c10f7-7694-4fdd-86ce-669947932edb-kube-api-access-z2p6c\") pod \"d44c10f7-7694-4fdd-86ce-669947932edb\" (UID: \"d44c10f7-7694-4fdd-86ce-669947932edb\") " Oct 30 01:46:31.732626 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:31.730354 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d44c10f7-7694-4fdd-86ce-669947932edb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d44c10f7-7694-4fdd-86ce-669947932edb-metrics-certs\") pod \"d44c10f7-7694-4fdd-86ce-669947932edb\" (UID: \"d44c10f7-7694-4fdd-86ce-669947932edb\") " Oct 30 01:46:31.732626 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:31.732337 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d44c10f7-7694-4fdd-86ce-669947932edb-openshift-service-ca" pod "d44c10f7-7694-4fdd-86ce-669947932edb" (UID: "d44c10f7-7694-4fdd-86ce-669947932edb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:46:31.836878 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:31.836839 6221 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d44c10f7-7694-4fdd-86ce-669947932edb-openshift-service-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:46:31.856555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:31.850738 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d44c10f7-7694-4fdd-86ce-669947932edb-kube-api-access-z2p6c" pod "d44c10f7-7694-4fdd-86ce-669947932edb" (UID: "d44c10f7-7694-4fdd-86ce-669947932edb"). InnerVolumeSpecName "kube-api-access-z2p6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:46:31.856555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:31.852653 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d44c10f7-7694-4fdd-86ce-669947932edb-maas-api-tls" pod "d44c10f7-7694-4fdd-86ce-669947932edb" (UID: "d44c10f7-7694-4fdd-86ce-669947932edb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:46:31.857019 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:31.856993 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d44c10f7-7694-4fdd-86ce-669947932edb-metrics-certs" pod "d44c10f7-7694-4fdd-86ce-669947932edb" (UID: "d44c10f7-7694-4fdd-86ce-669947932edb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:46:31.939936 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:31.939900 6221 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d44c10f7-7694-4fdd-86ce-669947932edb-metrics-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:46:31.939936 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:31.939936 6221 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d44c10f7-7694-4fdd-86ce-669947932edb-maas-api-tls\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:46:31.940191 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:31.939960 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2p6c\" (UniqueName: \"kubernetes.io/projected/d44c10f7-7694-4fdd-86ce-669947932edb-kube-api-access-z2p6c\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:46:32.132822 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:32.132724 6221 generic.go:356] "Generic (PLEG): container finished" podID="d44c10f7-7694-4fdd-86ce-669947932edb" containerID="14fc89219800f091e02610ce4ba6f579d858d093e4bf8f9364b8f42f7f5caebe" exitCode=0 Oct 30 01:46:32.132822 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:32.132783 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f197b1-5847f96fcd-4frnn" Oct 30 01:46:32.132822 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:32.132788 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f197b1-5847f96fcd-4frnn" event={"ID":"d44c10f7-7694-4fdd-86ce-669947932edb","Type":"ContainerDied","Data":"14fc89219800f091e02610ce4ba6f579d858d093e4bf8f9364b8f42f7f5caebe"} Oct 30 01:46:32.133108 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:32.132873 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f197b1-5847f96fcd-4frnn" event={"ID":"d44c10f7-7694-4fdd-86ce-669947932edb","Type":"ContainerDied","Data":"caa87dedf2264fd759d0a850a1825b0c9ac1cf4996a668e96628913769aca69a"} Oct 30 01:46:32.133108 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:32.132904 6221 scope.go:122] "RemoveContainer" containerID="14fc89219800f091e02610ce4ba6f579d858d093e4bf8f9364b8f42f7f5caebe" Oct 30 01:46:32.142940 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:32.142921 6221 scope.go:122] "RemoveContainer" containerID="14fc89219800f091e02610ce4ba6f579d858d093e4bf8f9364b8f42f7f5caebe" Oct 30 01:46:32.143201 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:46:32.143181 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14fc89219800f091e02610ce4ba6f579d858d093e4bf8f9364b8f42f7f5caebe\": container with ID starting with 14fc89219800f091e02610ce4ba6f579d858d093e4bf8f9364b8f42f7f5caebe not found: ID does not exist" containerID="14fc89219800f091e02610ce4ba6f579d858d093e4bf8f9364b8f42f7f5caebe" Oct 30 01:46:32.143262 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:32.143209 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14fc89219800f091e02610ce4ba6f579d858d093e4bf8f9364b8f42f7f5caebe"} err="failed to get container status \"14fc89219800f091e02610ce4ba6f579d858d093e4bf8f9364b8f42f7f5caebe\": rpc error: code = NotFound desc = could not find container \"14fc89219800f091e02610ce4ba6f579d858d093e4bf8f9364b8f42f7f5caebe\": container with ID starting with 14fc89219800f091e02610ce4ba6f579d858d093e4bf8f9364b8f42f7f5caebe not found: ID does not exist" Oct 30 01:46:32.184988 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:32.184965 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f197b1-5847f96fcd-4frnn"] Oct 30 01:46:32.196655 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:32.196628 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f197b1-5847f96fcd-4frnn"] Oct 30 01:46:32.548009 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:32.547970 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d44c10f7-7694-4fdd-86ce-669947932edb" path="/var/lib/kubelet/pods/d44c10f7-7694-4fdd-86ce-669947932edb/volumes" Oct 30 01:46:41.151621 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:41.150842 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-396f8e-88448749f-cwwq8" Oct 30 01:46:41.156739 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:41.156710 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-396f8e-c744b6894-4f4lh" Oct 30 01:46:41.237262 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:41.237209 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-396f8e-59798df9ff-qhchc"] Oct 30 01:46:41.238466 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:41.238433 6221 cpu_manager.go:414] "containerMap: removing container" podUID="d44c10f7-7694-4fdd-86ce-669947932edb" containerName="maas-api" Oct 30 01:46:41.238676 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:41.238657 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d44c10f7-7694-4fdd-86ce-669947932edb" containerName="maas-api" Oct 30 01:46:41.238970 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:41.238952 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="d44c10f7-7694-4fdd-86ce-669947932edb" containerName="maas-api" Oct 30 01:46:41.261558 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:41.248007 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-396f8e-59798df9ff-qhchc" Oct 30 01:46:41.282947 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:41.282892 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-396f8e-59798df9ff-qhchc"] Oct 30 01:46:41.300733 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:41.300682 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/348f8c54-8ced-40e2-846d-f551e78b3092-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-396f8e-59798df9ff-qhchc\" (UID: \"348f8c54-8ced-40e2-846d-f551e78b3092\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-396f8e-59798df9ff-qhchc" Oct 30 01:46:41.336028 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:41.302138 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-396f8e-serving-cert\"" Oct 30 01:46:41.336604 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:41.325674 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b774\" (UniqueName: \"kubernetes.io/projected/348f8c54-8ced-40e2-846d-f551e78b3092-kube-api-access-8b774\") pod \"maas-api-e2e-shared-a-w0-396f8e-59798df9ff-qhchc\" (UID: \"348f8c54-8ced-40e2-846d-f551e78b3092\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-396f8e-59798df9ff-qhchc" Oct 30 01:46:41.336847 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:41.336826 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/348f8c54-8ced-40e2-846d-f551e78b3092-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-396f8e-59798df9ff-qhchc\" (UID: \"348f8c54-8ced-40e2-846d-f551e78b3092\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-396f8e-59798df9ff-qhchc" Oct 30 01:46:41.337655 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:41.337631 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/348f8c54-8ced-40e2-846d-f551e78b3092-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-396f8e-59798df9ff-qhchc\" (UID: \"348f8c54-8ced-40e2-846d-f551e78b3092\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-396f8e-59798df9ff-qhchc" Oct 30 01:46:41.441823 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:41.441716 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b774\" (UniqueName: \"kubernetes.io/projected/348f8c54-8ced-40e2-846d-f551e78b3092-kube-api-access-8b774\") pod \"maas-api-e2e-shared-a-w0-396f8e-59798df9ff-qhchc\" (UID: \"348f8c54-8ced-40e2-846d-f551e78b3092\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-396f8e-59798df9ff-qhchc" Oct 30 01:46:41.442118 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:41.442078 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/348f8c54-8ced-40e2-846d-f551e78b3092-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-396f8e-59798df9ff-qhchc\" (UID: \"348f8c54-8ced-40e2-846d-f551e78b3092\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-396f8e-59798df9ff-qhchc" Oct 30 01:46:41.442390 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:41.442367 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/348f8c54-8ced-40e2-846d-f551e78b3092-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-396f8e-59798df9ff-qhchc\" (UID: \"348f8c54-8ced-40e2-846d-f551e78b3092\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-396f8e-59798df9ff-qhchc" Oct 30 01:46:41.442596 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:41.442578 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/348f8c54-8ced-40e2-846d-f551e78b3092-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-396f8e-59798df9ff-qhchc\" (UID: \"348f8c54-8ced-40e2-846d-f551e78b3092\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-396f8e-59798df9ff-qhchc" Oct 30 01:46:41.443019 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:46:41.442987 6221 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-396f8e-serving-cert: secret "maas-api-e2e-shared-a-w0-396f8e-serving-cert" not found Oct 30 01:46:41.443240 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:46:41.443226 6221 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/348f8c54-8ced-40e2-846d-f551e78b3092-maas-api-tls podName:348f8c54-8ced-40e2-846d-f551e78b3092 nodeName:}" failed. No retries permitted until 2026-10-30 01:46:41.94318352 +0000 UTC m=+1725.521470855 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/348f8c54-8ced-40e2-846d-f551e78b3092-maas-api-tls") pod "maas-api-e2e-shared-a-w0-396f8e-59798df9ff-qhchc" (UID: "348f8c54-8ced-40e2-846d-f551e78b3092") : secret "maas-api-e2e-shared-a-w0-396f8e-serving-cert" not found Oct 30 01:46:41.451905 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:41.451850 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/348f8c54-8ced-40e2-846d-f551e78b3092-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-396f8e-59798df9ff-qhchc\" (UID: \"348f8c54-8ced-40e2-846d-f551e78b3092\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-396f8e-59798df9ff-qhchc" Oct 30 01:46:41.472226 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:41.468525 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b774\" (UniqueName: \"kubernetes.io/projected/348f8c54-8ced-40e2-846d-f551e78b3092-kube-api-access-8b774\") pod \"maas-api-e2e-shared-a-w0-396f8e-59798df9ff-qhchc\" (UID: \"348f8c54-8ced-40e2-846d-f551e78b3092\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-396f8e-59798df9ff-qhchc" Oct 30 01:46:41.501559 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:41.494314 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/348f8c54-8ced-40e2-846d-f551e78b3092-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-396f8e-59798df9ff-qhchc\" (UID: \"348f8c54-8ced-40e2-846d-f551e78b3092\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-396f8e-59798df9ff-qhchc" Oct 30 01:46:41.966591 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:41.955568 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/348f8c54-8ced-40e2-846d-f551e78b3092-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-396f8e-59798df9ff-qhchc\" (UID: \"348f8c54-8ced-40e2-846d-f551e78b3092\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-396f8e-59798df9ff-qhchc" Oct 30 01:46:41.991245 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:41.989179 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/348f8c54-8ced-40e2-846d-f551e78b3092-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-396f8e-59798df9ff-qhchc\" (UID: \"348f8c54-8ced-40e2-846d-f551e78b3092\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-396f8e-59798df9ff-qhchc" Oct 30 01:46:42.259607 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:42.259177 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-396f8e-59798df9ff-qhchc" Oct 30 01:46:42.505557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:42.468881 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-0ae258/praxis-route-0ae258-kserve-fb65bffff-s8cc2" podUID="532d4bd3-5ece-42da-876f-496110860d4d" containerName="main" containerID="cri-o://26e018a2b9923dd6ecfab292bd09228395712887bec5204add579f413598c96c" gracePeriod=45 Oct 30 01:46:42.740556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:42.739893 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-0ae258/praxis-route-0ae258-kserve-fb65bffff-s8cc2" podUID="532d4bd3-5ece-42da-876f-496110860d4d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.127:8000/ready\": EOF" Oct 30 01:46:43.059556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:43.059383 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-396f8e-59798df9ff-qhchc"] Oct 30 01:46:43.306553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:43.303609 6221 generic.go:356] "Generic (PLEG): container finished" podID="532d4bd3-5ece-42da-876f-496110860d4d" containerID="26e018a2b9923dd6ecfab292bd09228395712887bec5204add579f413598c96c" exitCode=0 Oct 30 01:46:43.306553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:43.303780 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-0ae258/praxis-route-0ae258-kserve-fb65bffff-s8cc2" event={"ID":"532d4bd3-5ece-42da-876f-496110860d4d","Type":"ContainerDied","Data":"26e018a2b9923dd6ecfab292bd09228395712887bec5204add579f413598c96c"} Oct 30 01:46:43.336020 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:43.313658 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-396f8e-59798df9ff-qhchc" event={"ID":"348f8c54-8ced-40e2-846d-f551e78b3092","Type":"ContainerStarted","Data":"b0a53cdc3c0788ce832ce0b4944557f4500d624cf6e11e1497f15dcf4a4756a7"} Oct 30 01:46:43.482554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:43.471551 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-0ae258/praxis-route-0ae258-kserve-fb65bffff-s8cc2" podUID="532d4bd3-5ece-42da-876f-496110860d4d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.127:8000/ready\": dial tcp 10.134.0.127:8000: connect: connection refused" Oct 30 01:46:43.482554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:43.471683 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-0ae258/praxis-route-0ae258-kserve-fb65bffff-s8cc2" Oct 30 01:46:43.711382 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:43.711357 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-0ae258/praxis-route-0ae258-kserve-fb65bffff-s8cc2" Oct 30 01:46:43.785370 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:43.785336 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/532d4bd3-5ece-42da-876f-496110860d4d-kube-api-access-hg96r\" (UniqueName: \"kubernetes.io/projected/532d4bd3-5ece-42da-876f-496110860d4d-kube-api-access-hg96r\") pod \"532d4bd3-5ece-42da-876f-496110860d4d\" (UID: \"532d4bd3-5ece-42da-876f-496110860d4d\") " Oct 30 01:46:43.785590 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:43.785420 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/532d4bd3-5ece-42da-876f-496110860d4d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/532d4bd3-5ece-42da-876f-496110860d4d-tmp-dir\") pod \"532d4bd3-5ece-42da-876f-496110860d4d\" (UID: \"532d4bd3-5ece-42da-876f-496110860d4d\") " Oct 30 01:46:43.785672 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:43.785585 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/532d4bd3-5ece-42da-876f-496110860d4d-home\" (UniqueName: \"kubernetes.io/empty-dir/532d4bd3-5ece-42da-876f-496110860d4d-home\") pod \"532d4bd3-5ece-42da-876f-496110860d4d\" (UID: \"532d4bd3-5ece-42da-876f-496110860d4d\") " Oct 30 01:46:43.785672 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:43.785631 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/532d4bd3-5ece-42da-876f-496110860d4d-tls-certs\" (UniqueName: \"kubernetes.io/secret/532d4bd3-5ece-42da-876f-496110860d4d-tls-certs\") pod \"532d4bd3-5ece-42da-876f-496110860d4d\" (UID: \"532d4bd3-5ece-42da-876f-496110860d4d\") " Oct 30 01:46:43.785779 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:43.785674 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/532d4bd3-5ece-42da-876f-496110860d4d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/532d4bd3-5ece-42da-876f-496110860d4d-model-cache\") pod \"532d4bd3-5ece-42da-876f-496110860d4d\" (UID: \"532d4bd3-5ece-42da-876f-496110860d4d\") " Oct 30 01:46:43.785779 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:43.785720 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/532d4bd3-5ece-42da-876f-496110860d4d-tmp-dir" pod "532d4bd3-5ece-42da-876f-496110860d4d" (UID: "532d4bd3-5ece-42da-876f-496110860d4d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:46:43.785885 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:43.785849 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/532d4bd3-5ece-42da-876f-496110860d4d-home" pod "532d4bd3-5ece-42da-876f-496110860d4d" (UID: "532d4bd3-5ece-42da-876f-496110860d4d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:46:43.785940 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:43.785892 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/532d4bd3-5ece-42da-876f-496110860d4d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/532d4bd3-5ece-42da-876f-496110860d4d-dshm\") pod \"532d4bd3-5ece-42da-876f-496110860d4d\" (UID: \"532d4bd3-5ece-42da-876f-496110860d4d\") " Oct 30 01:46:43.786027 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:43.786008 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/532d4bd3-5ece-42da-876f-496110860d4d-model-cache" pod "532d4bd3-5ece-42da-876f-496110860d4d" (UID: "532d4bd3-5ece-42da-876f-496110860d4d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:46:43.786442 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:43.786415 6221 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/532d4bd3-5ece-42da-876f-496110860d4d-home\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:46:43.786593 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:43.786445 6221 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/532d4bd3-5ece-42da-876f-496110860d4d-model-cache\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:46:43.786593 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:43.786460 6221 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/532d4bd3-5ece-42da-876f-496110860d4d-tmp-dir\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:46:43.788139 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:43.788113 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/532d4bd3-5ece-42da-876f-496110860d4d-dshm" pod "532d4bd3-5ece-42da-876f-496110860d4d" (UID: "532d4bd3-5ece-42da-876f-496110860d4d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:46:43.788328 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:43.788305 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/532d4bd3-5ece-42da-876f-496110860d4d-tls-certs" pod "532d4bd3-5ece-42da-876f-496110860d4d" (UID: "532d4bd3-5ece-42da-876f-496110860d4d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:46:43.788328 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:43.788318 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/532d4bd3-5ece-42da-876f-496110860d4d-kube-api-access-hg96r" pod "532d4bd3-5ece-42da-876f-496110860d4d" (UID: "532d4bd3-5ece-42da-876f-496110860d4d"). InnerVolumeSpecName "kube-api-access-hg96r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:46:43.898472 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:43.898319 6221 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/532d4bd3-5ece-42da-876f-496110860d4d-tls-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:46:43.898472 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:43.898406 6221 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/532d4bd3-5ece-42da-876f-496110860d4d-dshm\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:46:43.898472 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:43.898424 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg96r\" (UniqueName: \"kubernetes.io/projected/532d4bd3-5ece-42da-876f-496110860d4d-kube-api-access-hg96r\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:46:44.322303 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:44.322263 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-396f8e-59798df9ff-qhchc" event={"ID":"348f8c54-8ced-40e2-846d-f551e78b3092","Type":"ContainerStarted","Data":"fe5130e9997b415e33591e0953ba9909def21256a0bf3f5749ddeeeab046ae6b"} Oct 30 01:46:44.322781 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:44.322335 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-396f8e-59798df9ff-qhchc" Oct 30 01:46:44.323787 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:44.323766 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-0ae258/praxis-route-0ae258-kserve-fb65bffff-s8cc2" Oct 30 01:46:44.323898 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:44.323786 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-0ae258/praxis-route-0ae258-kserve-fb65bffff-s8cc2" event={"ID":"532d4bd3-5ece-42da-876f-496110860d4d","Type":"ContainerDied","Data":"6496eccc167b45e6cea1697ced6e2a8a56df41100cd2875a3135afa1abed4bfd"} Oct 30 01:46:44.323898 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:44.323824 6221 scope.go:122] "RemoveContainer" containerID="26e018a2b9923dd6ecfab292bd09228395712887bec5204add579f413598c96c" Oct 30 01:46:44.369084 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:44.369032 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-396f8e-59798df9ff-qhchc" podStartSLOduration=2.927525676 podStartE2EDuration="3.369018946s" totalImagesPullingTime="441.49327ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:46:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:46:44.366792417 +0000 UTC m=+1727.945079757" watchObservedRunningTime="2026-10-30 01:46:44.369018946 +0000 UTC m=+1727.947306321" Oct 30 01:46:44.402734 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:44.402700 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-0ae258/praxis-route-0ae258-kserve-fb65bffff-s8cc2"] Oct 30 01:46:44.406215 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:44.406190 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-0ae258/praxis-route-0ae258-kserve-fb65bffff-s8cc2"] Oct 30 01:46:44.560699 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:44.560659 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="532d4bd3-5ece-42da-876f-496110860d4d" path="/var/lib/kubelet/pods/532d4bd3-5ece-42da-876f-496110860d4d/volumes" Oct 30 01:46:50.351326 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:50.350338 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-396f8e-59798df9ff-qhchc" Oct 30 01:46:51.495469 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:46:51.495437 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:46:52.705133 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:52.705100 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-8f68ab-5bf445cc7d-9b2mq"] Oct 30 01:46:52.705679 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:52.705309 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8f68ab-5bf445cc7d-9b2mq" podUID="9abd0232-8d93-4dfa-ad61-7b01ed80eba8" containerName="maas-api" containerID="cri-o://c4a4a81ca87831dd912e63113ed33c6e2d116b11d805a6aee2905be69a936532" gracePeriod=30 Oct 30 01:46:53.116087 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:53.116052 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8f68ab-5bf445cc7d-9b2mq" Oct 30 01:46:53.185921 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:53.185883 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9abd0232-8d93-4dfa-ad61-7b01ed80eba8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9abd0232-8d93-4dfa-ad61-7b01ed80eba8-metrics-certs\") pod \"9abd0232-8d93-4dfa-ad61-7b01ed80eba8\" (UID: \"9abd0232-8d93-4dfa-ad61-7b01ed80eba8\") " Oct 30 01:46:53.186327 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:53.186303 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9abd0232-8d93-4dfa-ad61-7b01ed80eba8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9abd0232-8d93-4dfa-ad61-7b01ed80eba8-openshift-service-ca\") pod \"9abd0232-8d93-4dfa-ad61-7b01ed80eba8\" (UID: \"9abd0232-8d93-4dfa-ad61-7b01ed80eba8\") " Oct 30 01:46:53.186583 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:53.186494 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9abd0232-8d93-4dfa-ad61-7b01ed80eba8-kube-api-access-64vv2\" (UniqueName: \"kubernetes.io/projected/9abd0232-8d93-4dfa-ad61-7b01ed80eba8-kube-api-access-64vv2\") pod \"9abd0232-8d93-4dfa-ad61-7b01ed80eba8\" (UID: \"9abd0232-8d93-4dfa-ad61-7b01ed80eba8\") " Oct 30 01:46:53.186742 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:53.186727 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9abd0232-8d93-4dfa-ad61-7b01ed80eba8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9abd0232-8d93-4dfa-ad61-7b01ed80eba8-maas-api-tls\") pod \"9abd0232-8d93-4dfa-ad61-7b01ed80eba8\" (UID: \"9abd0232-8d93-4dfa-ad61-7b01ed80eba8\") " Oct 30 01:46:53.188231 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:53.188189 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9abd0232-8d93-4dfa-ad61-7b01ed80eba8-openshift-service-ca" pod "9abd0232-8d93-4dfa-ad61-7b01ed80eba8" (UID: "9abd0232-8d93-4dfa-ad61-7b01ed80eba8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:46:53.190908 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:53.190881 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9abd0232-8d93-4dfa-ad61-7b01ed80eba8-metrics-certs" pod "9abd0232-8d93-4dfa-ad61-7b01ed80eba8" (UID: "9abd0232-8d93-4dfa-ad61-7b01ed80eba8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:46:53.191180 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:53.191159 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9abd0232-8d93-4dfa-ad61-7b01ed80eba8-kube-api-access-64vv2" pod "9abd0232-8d93-4dfa-ad61-7b01ed80eba8" (UID: "9abd0232-8d93-4dfa-ad61-7b01ed80eba8"). InnerVolumeSpecName "kube-api-access-64vv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:46:53.191357 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:53.191330 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9abd0232-8d93-4dfa-ad61-7b01ed80eba8-maas-api-tls" pod "9abd0232-8d93-4dfa-ad61-7b01ed80eba8" (UID: "9abd0232-8d93-4dfa-ad61-7b01ed80eba8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:46:53.315554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:53.290993 6221 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9abd0232-8d93-4dfa-ad61-7b01ed80eba8-metrics-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:46:53.315876 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:53.315849 6221 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9abd0232-8d93-4dfa-ad61-7b01ed80eba8-openshift-service-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:46:53.319633 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:53.319592 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64vv2\" (UniqueName: \"kubernetes.io/projected/9abd0232-8d93-4dfa-ad61-7b01ed80eba8-kube-api-access-64vv2\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:46:53.342555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:53.319856 6221 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9abd0232-8d93-4dfa-ad61-7b01ed80eba8-maas-api-tls\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:46:53.457657 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:53.457615 6221 generic.go:356] "Generic (PLEG): container finished" podID="9abd0232-8d93-4dfa-ad61-7b01ed80eba8" containerID="c4a4a81ca87831dd912e63113ed33c6e2d116b11d805a6aee2905be69a936532" exitCode=0 Oct 30 01:46:53.457831 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:53.457679 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8f68ab-5bf445cc7d-9b2mq" Oct 30 01:46:53.457831 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:53.457748 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8f68ab-5bf445cc7d-9b2mq" event={"ID":"9abd0232-8d93-4dfa-ad61-7b01ed80eba8","Type":"ContainerDied","Data":"c4a4a81ca87831dd912e63113ed33c6e2d116b11d805a6aee2905be69a936532"} Oct 30 01:46:53.457831 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:53.457824 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-8f68ab-5bf445cc7d-9b2mq" event={"ID":"9abd0232-8d93-4dfa-ad61-7b01ed80eba8","Type":"ContainerDied","Data":"a3d49d4d23ac30fddc1913199df3cdbf01a27ac0065c193ffa3d9b695456da3c"} Oct 30 01:46:53.457985 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:53.457857 6221 scope.go:122] "RemoveContainer" containerID="c4a4a81ca87831dd912e63113ed33c6e2d116b11d805a6aee2905be69a936532" Oct 30 01:46:53.468393 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:53.468370 6221 scope.go:122] "RemoveContainer" containerID="c4a4a81ca87831dd912e63113ed33c6e2d116b11d805a6aee2905be69a936532" Oct 30 01:46:53.468813 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:46:53.468789 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4a4a81ca87831dd912e63113ed33c6e2d116b11d805a6aee2905be69a936532\": container with ID starting with c4a4a81ca87831dd912e63113ed33c6e2d116b11d805a6aee2905be69a936532 not found: ID does not exist" containerID="c4a4a81ca87831dd912e63113ed33c6e2d116b11d805a6aee2905be69a936532" Oct 30 01:46:53.468866 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:53.468821 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4a4a81ca87831dd912e63113ed33c6e2d116b11d805a6aee2905be69a936532"} err="failed to get container status \"c4a4a81ca87831dd912e63113ed33c6e2d116b11d805a6aee2905be69a936532\": rpc error: code = NotFound desc = could not find container \"c4a4a81ca87831dd912e63113ed33c6e2d116b11d805a6aee2905be69a936532\": container with ID starting with c4a4a81ca87831dd912e63113ed33c6e2d116b11d805a6aee2905be69a936532 not found: ID does not exist" Oct 30 01:46:53.511924 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:53.511892 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-8f68ab-5bf445cc7d-9b2mq"] Oct 30 01:46:53.523759 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:53.523722 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-8f68ab-5bf445cc7d-9b2mq"] Oct 30 01:46:54.548082 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:54.548047 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9abd0232-8d93-4dfa-ad61-7b01ed80eba8" path="/var/lib/kubelet/pods/9abd0232-8d93-4dfa-ad61-7b01ed80eba8/volumes" Oct 30 01:46:57.241259 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:57.241227 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-abb30f-dfb544599-pjbvp"] Oct 30 01:46:57.242157 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:57.242137 6221 cpu_manager.go:414] "containerMap: removing container" podUID="9abd0232-8d93-4dfa-ad61-7b01ed80eba8" containerName="maas-api" Oct 30 01:46:57.242201 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:57.242160 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9abd0232-8d93-4dfa-ad61-7b01ed80eba8" containerName="maas-api" Oct 30 01:46:57.242240 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:57.242226 6221 cpu_manager.go:414] "containerMap: removing container" podUID="532d4bd3-5ece-42da-876f-496110860d4d" containerName="main" Oct 30 01:46:57.242240 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:57.242234 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="532d4bd3-5ece-42da-876f-496110860d4d" containerName="main" Oct 30 01:46:57.242383 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:57.242367 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="532d4bd3-5ece-42da-876f-496110860d4d" containerName="main" Oct 30 01:46:57.242383 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:57.242385 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="9abd0232-8d93-4dfa-ad61-7b01ed80eba8" containerName="maas-api" Oct 30 01:46:57.247136 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:57.247120 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-abb30f-dfb544599-pjbvp" Oct 30 01:46:57.263049 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:57.263019 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-abb30f-dfb544599-pjbvp"] Oct 30 01:46:57.272122 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:57.272094 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-abb30f\"" Oct 30 01:46:57.272476 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:57.272434 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-abb30f-dockercfg-nzr29\"" Oct 30 01:46:57.419555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:57.419496 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcbfae96-e276-4753-acb5-59ee55640aa8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-abb30f-dfb544599-pjbvp\" (UID: \"fcbfae96-e276-4753-acb5-59ee55640aa8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-abb30f-dfb544599-pjbvp" Oct 30 01:46:57.419863 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:57.419832 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9mmf\" (UniqueName: \"kubernetes.io/projected/fcbfae96-e276-4753-acb5-59ee55640aa8-kube-api-access-s9mmf\") pod \"payload-pre-processing-e2e-shared-a-w1-abb30f-dfb544599-pjbvp\" (UID: \"fcbfae96-e276-4753-acb5-59ee55640aa8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-abb30f-dfb544599-pjbvp" Oct 30 01:46:57.522784 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:57.522700 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcbfae96-e276-4753-acb5-59ee55640aa8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-abb30f-dfb544599-pjbvp\" (UID: \"fcbfae96-e276-4753-acb5-59ee55640aa8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-abb30f-dfb544599-pjbvp" Oct 30 01:46:57.522945 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:57.522804 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9mmf\" (UniqueName: \"kubernetes.io/projected/fcbfae96-e276-4753-acb5-59ee55640aa8-kube-api-access-s9mmf\") pod \"payload-pre-processing-e2e-shared-a-w1-abb30f-dfb544599-pjbvp\" (UID: \"fcbfae96-e276-4753-acb5-59ee55640aa8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-abb30f-dfb544599-pjbvp" Oct 30 01:46:57.524228 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:57.524198 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcbfae96-e276-4753-acb5-59ee55640aa8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-abb30f-dfb544599-pjbvp\" (UID: \"fcbfae96-e276-4753-acb5-59ee55640aa8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-abb30f-dfb544599-pjbvp" Oct 30 01:46:57.546422 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:57.545729 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-abb30f-549b6b6b86-lsn24"] Oct 30 01:46:57.565597 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:57.553429 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-abb30f-549b6b6b86-lsn24"] Oct 30 01:46:57.565597 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:57.553587 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-abb30f-549b6b6b86-lsn24" Oct 30 01:46:57.565597 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:57.557005 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9mmf\" (UniqueName: \"kubernetes.io/projected/fcbfae96-e276-4753-acb5-59ee55640aa8-kube-api-access-s9mmf\") pod \"payload-pre-processing-e2e-shared-a-w1-abb30f-dfb544599-pjbvp\" (UID: \"fcbfae96-e276-4753-acb5-59ee55640aa8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-abb30f-dfb544599-pjbvp" Oct 30 01:46:57.582071 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:57.582042 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-abb30f-dfb544599-pjbvp" Oct 30 01:46:57.628292 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:57.628259 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a577ca5d-a4d6-4f5c-8f0e-422290e6ec61-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-abb30f-549b6b6b86-lsn24\" (UID: \"a577ca5d-a4d6-4f5c-8f0e-422290e6ec61\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-abb30f-549b6b6b86-lsn24" Oct 30 01:46:57.628484 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:57.628322 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb57d\" (UniqueName: \"kubernetes.io/projected/a577ca5d-a4d6-4f5c-8f0e-422290e6ec61-kube-api-access-zb57d\") pod \"payload-processing-e2e-shared-a-w1-abb30f-549b6b6b86-lsn24\" (UID: \"a577ca5d-a4d6-4f5c-8f0e-422290e6ec61\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-abb30f-549b6b6b86-lsn24" Oct 30 01:46:57.730813 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:57.730766 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a577ca5d-a4d6-4f5c-8f0e-422290e6ec61-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-abb30f-549b6b6b86-lsn24\" (UID: \"a577ca5d-a4d6-4f5c-8f0e-422290e6ec61\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-abb30f-549b6b6b86-lsn24" Oct 30 01:46:57.731019 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:57.730857 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb57d\" (UniqueName: \"kubernetes.io/projected/a577ca5d-a4d6-4f5c-8f0e-422290e6ec61-kube-api-access-zb57d\") pod \"payload-processing-e2e-shared-a-w1-abb30f-549b6b6b86-lsn24\" (UID: \"a577ca5d-a4d6-4f5c-8f0e-422290e6ec61\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-abb30f-549b6b6b86-lsn24" Oct 30 01:46:57.731523 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:57.731497 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a577ca5d-a4d6-4f5c-8f0e-422290e6ec61-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-abb30f-549b6b6b86-lsn24\" (UID: \"a577ca5d-a4d6-4f5c-8f0e-422290e6ec61\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-abb30f-549b6b6b86-lsn24" Oct 30 01:46:57.750202 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:57.750169 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb57d\" (UniqueName: \"kubernetes.io/projected/a577ca5d-a4d6-4f5c-8f0e-422290e6ec61-kube-api-access-zb57d\") pod \"payload-processing-e2e-shared-a-w1-abb30f-549b6b6b86-lsn24\" (UID: \"a577ca5d-a4d6-4f5c-8f0e-422290e6ec61\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-abb30f-549b6b6b86-lsn24" Oct 30 01:46:57.913879 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:57.913839 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-abb30f-549b6b6b86-lsn24" Oct 30 01:46:57.939126 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:57.939091 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-abb30f-dfb544599-pjbvp"] Oct 30 01:46:57.942189 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:46:57.942159 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcbfae96_e276_4753_acb5_59ee55640aa8.slice/crio-3cda9e6ad6eda8b53adbc7d94f677d72479455c80b069d5f7c8475b70688981d WatchSource:0}: Error finding container 3cda9e6ad6eda8b53adbc7d94f677d72479455c80b069d5f7c8475b70688981d: Status 404 returned error can't find the container with id 3cda9e6ad6eda8b53adbc7d94f677d72479455c80b069d5f7c8475b70688981d Oct 30 01:46:58.219260 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:58.219234 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-abb30f-549b6b6b86-lsn24"] Oct 30 01:46:58.222039 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:46:58.222007 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda577ca5d_a4d6_4f5c_8f0e_422290e6ec61.slice/crio-eb152ae76d4239d668fd7080e30562d3adf71cbae4cde78d3babf155bfbf995b WatchSource:0}: Error finding container eb152ae76d4239d668fd7080e30562d3adf71cbae4cde78d3babf155bfbf995b: Status 404 returned error can't find the container with id eb152ae76d4239d668fd7080e30562d3adf71cbae4cde78d3babf155bfbf995b Oct 30 01:46:58.569598 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:58.569231 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-abb30f-549b6b6b86-lsn24" event={"ID":"a577ca5d-a4d6-4f5c-8f0e-422290e6ec61","Type":"ContainerStarted","Data":"8116d6c60940e8edb4dab1ab98cdf66941cfb3f7cd0599a1d45347737f6a09de"} Oct 30 01:46:58.569598 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:58.569397 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-abb30f-549b6b6b86-lsn24" Oct 30 01:46:58.569598 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:58.569420 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-abb30f-549b6b6b86-lsn24" event={"ID":"a577ca5d-a4d6-4f5c-8f0e-422290e6ec61","Type":"ContainerStarted","Data":"eb152ae76d4239d668fd7080e30562d3adf71cbae4cde78d3babf155bfbf995b"} Oct 30 01:46:58.571013 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:58.570989 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-abb30f-dfb544599-pjbvp" event={"ID":"fcbfae96-e276-4753-acb5-59ee55640aa8","Type":"ContainerStarted","Data":"fdfa2efd1150de51a574ed459f115172655639f9771909ca0ceb8ddaf372fb67"} Oct 30 01:46:58.571121 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:58.571032 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-abb30f-dfb544599-pjbvp" event={"ID":"fcbfae96-e276-4753-acb5-59ee55640aa8","Type":"ContainerStarted","Data":"3cda9e6ad6eda8b53adbc7d94f677d72479455c80b069d5f7c8475b70688981d"} Oct 30 01:46:58.571121 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:58.571068 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-abb30f-dfb544599-pjbvp" Oct 30 01:46:58.610849 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:58.610794 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-abb30f-549b6b6b86-lsn24" podStartSLOduration=1.610780058 podStartE2EDuration="1.610780058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:46:58.609052689 +0000 UTC m=+1742.187340038" watchObservedRunningTime="2026-10-30 01:46:58.610780058 +0000 UTC m=+1742.189067394" Oct 30 01:46:58.651994 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:46:58.651938 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-abb30f-dfb544599-pjbvp" podStartSLOduration=1.651920614 podStartE2EDuration="1.651920614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:46:58.649285072 +0000 UTC m=+1742.227572411" watchObservedRunningTime="2026-10-30 01:46:58.651920614 +0000 UTC m=+1742.230207950" Oct 30 01:47:00.359762 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:00.359719 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz"] Oct 30 01:47:00.365031 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:00.365005 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" Oct 30 01:47:00.368360 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:00.368326 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqm52\" (UniqueName: \"kubernetes.io/projected/fd152ecb-4d3c-494c-83c1-182f018a4db1-kube-api-access-rqm52\") pod \"e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz\" (UID: \"fd152ecb-4d3c-494c-83c1-182f018a4db1\") " pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" Oct 30 01:47:00.368580 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:00.368476 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd152ecb-4d3c-494c-83c1-182f018a4db1-istio-data\") pod \"e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz\" (UID: \"fd152ecb-4d3c-494c-83c1-182f018a4db1\") " pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" Oct 30 01:47:00.368670 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:00.368589 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd152ecb-4d3c-494c-83c1-182f018a4db1-istio-podinfo\") pod \"e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz\" (UID: \"fd152ecb-4d3c-494c-83c1-182f018a4db1\") " pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" Oct 30 01:47:00.368670 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:00.368627 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd152ecb-4d3c-494c-83c1-182f018a4db1-istio-envoy\") pod \"e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz\" (UID: \"fd152ecb-4d3c-494c-83c1-182f018a4db1\") " pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" Oct 30 01:47:00.368788 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:00.368692 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd152ecb-4d3c-494c-83c1-182f018a4db1-credential-socket\") pod \"e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz\" (UID: \"fd152ecb-4d3c-494c-83c1-182f018a4db1\") " pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" Oct 30 01:47:00.368788 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:00.368773 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd152ecb-4d3c-494c-83c1-182f018a4db1-workload-certs\") pod \"e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz\" (UID: \"fd152ecb-4d3c-494c-83c1-182f018a4db1\") " pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" Oct 30 01:47:00.368904 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:00.368828 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd152ecb-4d3c-494c-83c1-182f018a4db1-workload-socket\") pod \"e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz\" (UID: \"fd152ecb-4d3c-494c-83c1-182f018a4db1\") " pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" Oct 30 01:47:00.368904 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:00.368879 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd152ecb-4d3c-494c-83c1-182f018a4db1-istiod-ca-cert\") pod \"e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz\" (UID: \"fd152ecb-4d3c-494c-83c1-182f018a4db1\") " pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" Oct 30 01:47:00.369011 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:00.368920 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd152ecb-4d3c-494c-83c1-182f018a4db1-istio-token\") pod \"e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz\" (UID: \"fd152ecb-4d3c-494c-83c1-182f018a4db1\") " pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" Oct 30 01:47:00.394002 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:00.393970 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz"] Oct 30 01:47:00.395154 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:00.395130 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-b6b7fc-openshift-default-dockercfg-mzlvh\"" Oct 30 01:47:00.470695 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:00.470453 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd152ecb-4d3c-494c-83c1-182f018a4db1-istio-data\") pod \"e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz\" (UID: \"fd152ecb-4d3c-494c-83c1-182f018a4db1\") " pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" Oct 30 01:47:00.470695 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:00.470575 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd152ecb-4d3c-494c-83c1-182f018a4db1-istio-podinfo\") pod \"e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz\" (UID: \"fd152ecb-4d3c-494c-83c1-182f018a4db1\") " pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" Oct 30 01:47:00.470695 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:00.470615 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd152ecb-4d3c-494c-83c1-182f018a4db1-istio-envoy\") pod \"e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz\" (UID: \"fd152ecb-4d3c-494c-83c1-182f018a4db1\") " pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" Oct 30 01:47:00.470695 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:00.470681 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd152ecb-4d3c-494c-83c1-182f018a4db1-credential-socket\") pod \"e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz\" (UID: \"fd152ecb-4d3c-494c-83c1-182f018a4db1\") " pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" Oct 30 01:47:00.471039 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:00.470761 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd152ecb-4d3c-494c-83c1-182f018a4db1-workload-certs\") pod \"e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz\" (UID: \"fd152ecb-4d3c-494c-83c1-182f018a4db1\") " pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" Oct 30 01:47:00.471039 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:00.470814 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd152ecb-4d3c-494c-83c1-182f018a4db1-workload-socket\") pod \"e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz\" (UID: \"fd152ecb-4d3c-494c-83c1-182f018a4db1\") " pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" Oct 30 01:47:00.471039 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:00.470865 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd152ecb-4d3c-494c-83c1-182f018a4db1-istiod-ca-cert\") pod \"e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz\" (UID: \"fd152ecb-4d3c-494c-83c1-182f018a4db1\") " pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" Oct 30 01:47:00.471039 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:00.470903 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd152ecb-4d3c-494c-83c1-182f018a4db1-istio-token\") pod \"e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz\" (UID: \"fd152ecb-4d3c-494c-83c1-182f018a4db1\") " pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" Oct 30 01:47:00.471039 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:00.470965 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqm52\" (UniqueName: \"kubernetes.io/projected/fd152ecb-4d3c-494c-83c1-182f018a4db1-kube-api-access-rqm52\") pod \"e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz\" (UID: \"fd152ecb-4d3c-494c-83c1-182f018a4db1\") " pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" Oct 30 01:47:00.471317 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:00.471128 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd152ecb-4d3c-494c-83c1-182f018a4db1-istio-data\") pod \"e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz\" (UID: \"fd152ecb-4d3c-494c-83c1-182f018a4db1\") " pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" Oct 30 01:47:00.471317 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:00.471249 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd152ecb-4d3c-494c-83c1-182f018a4db1-workload-certs\") pod \"e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz\" (UID: \"fd152ecb-4d3c-494c-83c1-182f018a4db1\") " pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" Oct 30 01:47:00.471636 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:00.471564 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd152ecb-4d3c-494c-83c1-182f018a4db1-credential-socket\") pod \"e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz\" (UID: \"fd152ecb-4d3c-494c-83c1-182f018a4db1\") " pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" Oct 30 01:47:00.472422 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:00.471752 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd152ecb-4d3c-494c-83c1-182f018a4db1-workload-socket\") pod \"e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz\" (UID: \"fd152ecb-4d3c-494c-83c1-182f018a4db1\") " pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" Oct 30 01:47:00.472422 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:00.471852 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd152ecb-4d3c-494c-83c1-182f018a4db1-istiod-ca-cert\") pod \"e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz\" (UID: \"fd152ecb-4d3c-494c-83c1-182f018a4db1\") " pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" Oct 30 01:47:00.508941 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:00.507234 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd152ecb-4d3c-494c-83c1-182f018a4db1-istio-podinfo\") pod \"e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz\" (UID: \"fd152ecb-4d3c-494c-83c1-182f018a4db1\") " pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" Oct 30 01:47:00.508941 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:00.507451 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd152ecb-4d3c-494c-83c1-182f018a4db1-istio-envoy\") pod \"e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz\" (UID: \"fd152ecb-4d3c-494c-83c1-182f018a4db1\") " pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" Oct 30 01:47:00.632612 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:00.588319 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd152ecb-4d3c-494c-83c1-182f018a4db1-istio-token\") pod \"e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz\" (UID: \"fd152ecb-4d3c-494c-83c1-182f018a4db1\") " pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" Oct 30 01:47:00.632612 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:00.624346 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqm52\" (UniqueName: \"kubernetes.io/projected/fd152ecb-4d3c-494c-83c1-182f018a4db1-kube-api-access-rqm52\") pod \"e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz\" (UID: \"fd152ecb-4d3c-494c-83c1-182f018a4db1\") " pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" Oct 30 01:47:00.686068 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:00.686029 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" Oct 30 01:47:01.075823 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:01.075789 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz"] Oct 30 01:47:01.078366 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:47:01.078335 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd152ecb_4d3c_494c_83c1_182f018a4db1.slice/crio-652f9af5779cb77be07e0810ceec9b352202b3854a9c6388b6f308b7e7618d12 WatchSource:0}: Error finding container 652f9af5779cb77be07e0810ceec9b352202b3854a9c6388b6f308b7e7618d12: Status 404 returned error can't find the container with id 652f9af5779cb77be07e0810ceec9b352202b3854a9c6388b6f308b7e7618d12 Oct 30 01:47:01.080998 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:01.080963 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:47:01.081094 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:01.081037 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:47:01.081094 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:01.081067 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:47:01.146193 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:01.146147 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q"] Oct 30 01:47:01.146429 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:01.146400 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" podUID="cbbfc7ea-d855-4b77-b484-b0b316f40963" containerName="istio-proxy" containerID="cri-o://09281d86ec47149a0eef45e57814712b7d5d7dbc05a8e778b588778dd32984f1" gracePeriod=30 Oct 30 01:47:01.665996 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:01.665955 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" event={"ID":"fd152ecb-4d3c-494c-83c1-182f018a4db1","Type":"ContainerStarted","Data":"39aaa2dd6656f6d14ad55ea410fb92f75c423e6c041af570bed8c4a989d616b1"} Oct 30 01:47:01.666422 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:01.666017 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" event={"ID":"fd152ecb-4d3c-494c-83c1-182f018a4db1","Type":"ContainerStarted","Data":"652f9af5779cb77be07e0810ceec9b352202b3854a9c6388b6f308b7e7618d12"} Oct 30 01:47:01.688721 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:01.688690 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" Oct 30 01:47:01.711489 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:01.711436 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" podStartSLOduration=1.711420752 podStartE2EDuration="1.711420752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:47:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:47:01.710727969 +0000 UTC m=+1745.289015305" watchObservedRunningTime="2026-10-30 01:47:01.711420752 +0000 UTC m=+1745.289708088" Oct 30 01:47:02.694950 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:02.694919 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" Oct 30 01:47:03.276103 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:03.276068 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:47:03.679474 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:03.679442 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" Oct 30 01:47:03.680501 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:03.680484 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" Oct 30 01:47:06.412551 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.412219 6221 patch_prober.go:29] interesting pod/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 01:47:06.412551 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.412287 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" podUID="cbbfc7ea-d855-4b77-b484-b0b316f40963" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 01:47:06.570371 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.570339 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b6b7fc-546d4d784c-6cjjd"] Oct 30 01:47:06.574709 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.574690 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b6b7fc-546d4d784c-6cjjd" Oct 30 01:47:06.575932 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.575915 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" Oct 30 01:47:06.592859 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.592835 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b6b7fc-546d4d784c-6cjjd"] Oct 30 01:47:06.601999 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.601953 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-b6b7fc\"" Oct 30 01:47:06.602133 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.602074 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-b6b7fc-dockercfg-4jt42\"" Oct 30 01:47:06.637584 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.621924 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbbfc7ea-d855-4b77-b484-b0b316f40963-kube-api-access-7lzqt\" (UniqueName: \"kubernetes.io/projected/cbbfc7ea-d855-4b77-b484-b0b316f40963-kube-api-access-7lzqt\") pod \"cbbfc7ea-d855-4b77-b484-b0b316f40963\" (UID: \"cbbfc7ea-d855-4b77-b484-b0b316f40963\") " Oct 30 01:47:06.637584 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.621985 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cbbfc7ea-d855-4b77-b484-b0b316f40963-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbbfc7ea-d855-4b77-b484-b0b316f40963-istio-podinfo\") pod \"cbbfc7ea-d855-4b77-b484-b0b316f40963\" (UID: \"cbbfc7ea-d855-4b77-b484-b0b316f40963\") " Oct 30 01:47:06.637584 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.622458 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbbfc7ea-d855-4b77-b484-b0b316f40963-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbbfc7ea-d855-4b77-b484-b0b316f40963-credential-socket\") pod \"cbbfc7ea-d855-4b77-b484-b0b316f40963\" (UID: \"cbbfc7ea-d855-4b77-b484-b0b316f40963\") " Oct 30 01:47:06.637584 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.630633 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbbfc7ea-d855-4b77-b484-b0b316f40963-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbbfc7ea-d855-4b77-b484-b0b316f40963-istio-envoy\") pod \"cbbfc7ea-d855-4b77-b484-b0b316f40963\" (UID: \"cbbfc7ea-d855-4b77-b484-b0b316f40963\") " Oct 30 01:47:06.637584 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.630749 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbbfc7ea-d855-4b77-b484-b0b316f40963-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbbfc7ea-d855-4b77-b484-b0b316f40963-istio-data\") pod \"cbbfc7ea-d855-4b77-b484-b0b316f40963\" (UID: \"cbbfc7ea-d855-4b77-b484-b0b316f40963\") " Oct 30 01:47:06.637584 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.630820 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbbfc7ea-d855-4b77-b484-b0b316f40963-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbbfc7ea-d855-4b77-b484-b0b316f40963-workload-certs\") pod \"cbbfc7ea-d855-4b77-b484-b0b316f40963\" (UID: \"cbbfc7ea-d855-4b77-b484-b0b316f40963\") " Oct 30 01:47:06.637584 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.630974 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbbfc7ea-d855-4b77-b484-b0b316f40963-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbbfc7ea-d855-4b77-b484-b0b316f40963-workload-socket\") pod \"cbbfc7ea-d855-4b77-b484-b0b316f40963\" (UID: \"cbbfc7ea-d855-4b77-b484-b0b316f40963\") " Oct 30 01:47:06.637584 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.631011 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbbfc7ea-d855-4b77-b484-b0b316f40963-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbbfc7ea-d855-4b77-b484-b0b316f40963-istiod-ca-cert\") pod \"cbbfc7ea-d855-4b77-b484-b0b316f40963\" (UID: \"cbbfc7ea-d855-4b77-b484-b0b316f40963\") " Oct 30 01:47:06.637584 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.631119 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbbfc7ea-d855-4b77-b484-b0b316f40963-istio-token\" (UniqueName: \"kubernetes.io/projected/cbbfc7ea-d855-4b77-b484-b0b316f40963-istio-token\") pod \"cbbfc7ea-d855-4b77-b484-b0b316f40963\" (UID: \"cbbfc7ea-d855-4b77-b484-b0b316f40963\") " Oct 30 01:47:06.637584 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.631612 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9061008c-04d0-42d9-96ae-a8ebe16c54b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-b6b7fc-546d4d784c-6cjjd\" (UID: \"9061008c-04d0-42d9-96ae-a8ebe16c54b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b6b7fc-546d4d784c-6cjjd" Oct 30 01:47:06.637584 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.631883 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ccjd\" (UniqueName: \"kubernetes.io/projected/9061008c-04d0-42d9-96ae-a8ebe16c54b7-kube-api-access-6ccjd\") pod \"payload-pre-processing-e2e-shared-a-w2-b6b7fc-546d4d784c-6cjjd\" (UID: \"9061008c-04d0-42d9-96ae-a8ebe16c54b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b6b7fc-546d4d784c-6cjjd" Oct 30 01:47:06.637584 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.624215 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbbfc7ea-d855-4b77-b484-b0b316f40963-credential-socket" pod "cbbfc7ea-d855-4b77-b484-b0b316f40963" (UID: "cbbfc7ea-d855-4b77-b484-b0b316f40963"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:47:06.659783 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.653370 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbbfc7ea-d855-4b77-b484-b0b316f40963-workload-socket" pod "cbbfc7ea-d855-4b77-b484-b0b316f40963" (UID: "cbbfc7ea-d855-4b77-b484-b0b316f40963"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:47:06.659783 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.653637 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbbfc7ea-d855-4b77-b484-b0b316f40963-istio-data" pod "cbbfc7ea-d855-4b77-b484-b0b316f40963" (UID: "cbbfc7ea-d855-4b77-b484-b0b316f40963"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:47:06.659783 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.653836 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbbfc7ea-d855-4b77-b484-b0b316f40963-workload-certs" pod "cbbfc7ea-d855-4b77-b484-b0b316f40963" (UID: "cbbfc7ea-d855-4b77-b484-b0b316f40963"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:47:06.659783 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.658130 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbbfc7ea-d855-4b77-b484-b0b316f40963-istiod-ca-cert" pod "cbbfc7ea-d855-4b77-b484-b0b316f40963" (UID: "cbbfc7ea-d855-4b77-b484-b0b316f40963"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:47:06.659783 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.658695 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbbfc7ea-d855-4b77-b484-b0b316f40963-istio-envoy" pod "cbbfc7ea-d855-4b77-b484-b0b316f40963" (UID: "cbbfc7ea-d855-4b77-b484-b0b316f40963"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:47:06.678895 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.673627 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b6b7fc-68dff5d4d-295gs"] Oct 30 01:47:06.678895 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.674680 6221 cpu_manager.go:414] "containerMap: removing container" podUID="cbbfc7ea-d855-4b77-b484-b0b316f40963" containerName="istio-proxy" Oct 30 01:47:06.678895 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.674702 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbbfc7ea-d855-4b77-b484-b0b316f40963" containerName="istio-proxy" Oct 30 01:47:06.678895 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.674914 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbbfc7ea-d855-4b77-b484-b0b316f40963" containerName="istio-proxy" Oct 30 01:47:06.702878 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.693876 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b6b7fc-68dff5d4d-295gs" Oct 30 01:47:06.704350 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.704263 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b6b7fc-68dff5d4d-295gs"] Oct 30 01:47:06.830608 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.764394 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvm5m\" (UniqueName: \"kubernetes.io/projected/a36fc050-879a-4182-ac44-6d3ad2ff189e-kube-api-access-fvm5m\") pod \"payload-processing-e2e-shared-a-w2-b6b7fc-68dff5d4d-295gs\" (UID: \"a36fc050-879a-4182-ac44-6d3ad2ff189e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b6b7fc-68dff5d4d-295gs" Oct 30 01:47:06.830608 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.764523 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9061008c-04d0-42d9-96ae-a8ebe16c54b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-b6b7fc-546d4d784c-6cjjd\" (UID: \"9061008c-04d0-42d9-96ae-a8ebe16c54b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b6b7fc-546d4d784c-6cjjd" Oct 30 01:47:06.830608 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.768078 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9061008c-04d0-42d9-96ae-a8ebe16c54b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-b6b7fc-546d4d784c-6cjjd\" (UID: \"9061008c-04d0-42d9-96ae-a8ebe16c54b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b6b7fc-546d4d784c-6cjjd" Oct 30 01:47:06.830608 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.798269 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbbfc7ea-d855-4b77-b484-b0b316f40963-istio-token" pod "cbbfc7ea-d855-4b77-b484-b0b316f40963" (UID: "cbbfc7ea-d855-4b77-b484-b0b316f40963"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:47:06.830608 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.764711 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a36fc050-879a-4182-ac44-6d3ad2ff189e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-b6b7fc-68dff5d4d-295gs\" (UID: \"a36fc050-879a-4182-ac44-6d3ad2ff189e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b6b7fc-68dff5d4d-295gs" Oct 30 01:47:06.830608 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.810390 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ccjd\" (UniqueName: \"kubernetes.io/projected/9061008c-04d0-42d9-96ae-a8ebe16c54b7-kube-api-access-6ccjd\") pod \"payload-pre-processing-e2e-shared-a-w2-b6b7fc-546d4d784c-6cjjd\" (UID: \"9061008c-04d0-42d9-96ae-a8ebe16c54b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b6b7fc-546d4d784c-6cjjd" Oct 30 01:47:06.830608 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.810617 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbbfc7ea-d855-4b77-b484-b0b316f40963-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:47:06.830608 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.810632 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbbfc7ea-d855-4b77-b484-b0b316f40963-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:47:06.830608 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.810651 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbbfc7ea-d855-4b77-b484-b0b316f40963-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:47:06.830608 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.810662 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbbfc7ea-d855-4b77-b484-b0b316f40963-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:47:06.830608 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.810673 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbbfc7ea-d855-4b77-b484-b0b316f40963-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:47:06.830608 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.810685 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbbfc7ea-d855-4b77-b484-b0b316f40963-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:47:06.830608 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.810696 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbbfc7ea-d855-4b77-b484-b0b316f40963-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:47:06.834069 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.834002 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ccjd\" (UniqueName: \"kubernetes.io/projected/9061008c-04d0-42d9-96ae-a8ebe16c54b7-kube-api-access-6ccjd\") pod \"payload-pre-processing-e2e-shared-a-w2-b6b7fc-546d4d784c-6cjjd\" (UID: \"9061008c-04d0-42d9-96ae-a8ebe16c54b7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b6b7fc-546d4d784c-6cjjd" Oct 30 01:47:06.861689 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.861650 6221 generic.go:356] "Generic (PLEG): container finished" podID="cbbfc7ea-d855-4b77-b484-b0b316f40963" containerID="09281d86ec47149a0eef45e57814712b7d5d7dbc05a8e778b588778dd32984f1" exitCode=0 Oct 30 01:47:06.861938 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.861912 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" event={"ID":"cbbfc7ea-d855-4b77-b484-b0b316f40963","Type":"ContainerDied","Data":"09281d86ec47149a0eef45e57814712b7d5d7dbc05a8e778b588778dd32984f1"} Oct 30 01:47:06.862911 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.862104 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" event={"ID":"cbbfc7ea-d855-4b77-b484-b0b316f40963","Type":"ContainerDied","Data":"c28c56adc9744b7920a9d2ab5cb9eadd5d38bc70f9fc811e5de60c55b8e0d815"} Oct 30 01:47:06.862911 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.862150 6221 scope.go:122] "RemoveContainer" containerID="09281d86ec47149a0eef45e57814712b7d5d7dbc05a8e778b588778dd32984f1" Oct 30 01:47:06.862911 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.862389 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q" Oct 30 01:47:06.869026 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.868990 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cbbfc7ea-d855-4b77-b484-b0b316f40963-istio-podinfo" pod "cbbfc7ea-d855-4b77-b484-b0b316f40963" (UID: "cbbfc7ea-d855-4b77-b484-b0b316f40963"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:47:06.880139 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.880022 6221 scope.go:122] "RemoveContainer" containerID="443dc49ae30b6ac2faf2336d59c3ed039cae65ec6c66df5f05ed79e88a85a8cc" Oct 30 01:47:06.880967 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.880184 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbbfc7ea-d855-4b77-b484-b0b316f40963-kube-api-access-7lzqt" pod "cbbfc7ea-d855-4b77-b484-b0b316f40963" (UID: "cbbfc7ea-d855-4b77-b484-b0b316f40963"). InnerVolumeSpecName "kube-api-access-7lzqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:47:06.888260 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.888232 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b6b7fc-546d4d784c-6cjjd" Oct 30 01:47:06.893137 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.892681 6221 scope.go:122] "RemoveContainer" containerID="09281d86ec47149a0eef45e57814712b7d5d7dbc05a8e778b588778dd32984f1" Oct 30 01:47:06.893511 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:47:06.893488 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09281d86ec47149a0eef45e57814712b7d5d7dbc05a8e778b588778dd32984f1\": container with ID starting with 09281d86ec47149a0eef45e57814712b7d5d7dbc05a8e778b588778dd32984f1 not found: ID does not exist" containerID="09281d86ec47149a0eef45e57814712b7d5d7dbc05a8e778b588778dd32984f1" Oct 30 01:47:06.893626 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.893525 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09281d86ec47149a0eef45e57814712b7d5d7dbc05a8e778b588778dd32984f1"} err="failed to get container status \"09281d86ec47149a0eef45e57814712b7d5d7dbc05a8e778b588778dd32984f1\": rpc error: code = NotFound desc = could not find container \"09281d86ec47149a0eef45e57814712b7d5d7dbc05a8e778b588778dd32984f1\": container with ID starting with 09281d86ec47149a0eef45e57814712b7d5d7dbc05a8e778b588778dd32984f1 not found: ID does not exist" Oct 30 01:47:06.893626 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.893562 6221 scope.go:122] "RemoveContainer" containerID="443dc49ae30b6ac2faf2336d59c3ed039cae65ec6c66df5f05ed79e88a85a8cc" Oct 30 01:47:06.893878 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:47:06.893860 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"443dc49ae30b6ac2faf2336d59c3ed039cae65ec6c66df5f05ed79e88a85a8cc\": container with ID starting with 443dc49ae30b6ac2faf2336d59c3ed039cae65ec6c66df5f05ed79e88a85a8cc not found: ID does not exist" containerID="443dc49ae30b6ac2faf2336d59c3ed039cae65ec6c66df5f05ed79e88a85a8cc" Oct 30 01:47:06.893943 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.893886 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"443dc49ae30b6ac2faf2336d59c3ed039cae65ec6c66df5f05ed79e88a85a8cc"} err="failed to get container status \"443dc49ae30b6ac2faf2336d59c3ed039cae65ec6c66df5f05ed79e88a85a8cc\": rpc error: code = NotFound desc = could not find container \"443dc49ae30b6ac2faf2336d59c3ed039cae65ec6c66df5f05ed79e88a85a8cc\": container with ID starting with 443dc49ae30b6ac2faf2336d59c3ed039cae65ec6c66df5f05ed79e88a85a8cc not found: ID does not exist" Oct 30 01:47:06.912113 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.912081 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a36fc050-879a-4182-ac44-6d3ad2ff189e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-b6b7fc-68dff5d4d-295gs\" (UID: \"a36fc050-879a-4182-ac44-6d3ad2ff189e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b6b7fc-68dff5d4d-295gs" Oct 30 01:47:06.912237 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.912186 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvm5m\" (UniqueName: \"kubernetes.io/projected/a36fc050-879a-4182-ac44-6d3ad2ff189e-kube-api-access-fvm5m\") pod \"payload-processing-e2e-shared-a-w2-b6b7fc-68dff5d4d-295gs\" (UID: \"a36fc050-879a-4182-ac44-6d3ad2ff189e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b6b7fc-68dff5d4d-295gs" Oct 30 01:47:06.912380 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.912364 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lzqt\" (UniqueName: \"kubernetes.io/projected/cbbfc7ea-d855-4b77-b484-b0b316f40963-kube-api-access-7lzqt\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:47:06.912418 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.912388 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbbfc7ea-d855-4b77-b484-b0b316f40963-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:47:06.912872 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.912854 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a36fc050-879a-4182-ac44-6d3ad2ff189e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-b6b7fc-68dff5d4d-295gs\" (UID: \"a36fc050-879a-4182-ac44-6d3ad2ff189e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b6b7fc-68dff5d4d-295gs" Oct 30 01:47:06.931847 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:06.931762 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvm5m\" (UniqueName: \"kubernetes.io/projected/a36fc050-879a-4182-ac44-6d3ad2ff189e-kube-api-access-fvm5m\") pod \"payload-processing-e2e-shared-a-w2-b6b7fc-68dff5d4d-295gs\" (UID: \"a36fc050-879a-4182-ac44-6d3ad2ff189e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b6b7fc-68dff5d4d-295gs" Oct 30 01:47:07.148983 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:07.148899 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b6b7fc-68dff5d4d-295gs" Oct 30 01:47:07.192221 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:07.192193 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b6b7fc-546d4d784c-6cjjd"] Oct 30 01:47:07.193888 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:47:07.193861 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9061008c_04d0_42d9_96ae_a8ebe16c54b7.slice/crio-46b4ab5bbd837a0303bf3e5882c90314828809b5ad3469b1c57dd27952530154 WatchSource:0}: Error finding container 46b4ab5bbd837a0303bf3e5882c90314828809b5ad3469b1c57dd27952530154: Status 404 returned error can't find the container with id 46b4ab5bbd837a0303bf3e5882c90314828809b5ad3469b1c57dd27952530154 Oct 30 01:47:07.212231 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:07.212204 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q"] Oct 30 01:47:07.216763 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:07.216741 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-8f68ab-openshift-default-79487fd489-prk6q"] Oct 30 01:47:07.459735 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:07.459710 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b6b7fc-68dff5d4d-295gs"] Oct 30 01:47:07.462558 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:47:07.462504 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda36fc050_879a_4182_ac44_6d3ad2ff189e.slice/crio-5423a8546089615a9991bfdeab5345de774e5012104e25f4622feec1c58b19dc WatchSource:0}: Error finding container 5423a8546089615a9991bfdeab5345de774e5012104e25f4622feec1c58b19dc: Status 404 returned error can't find the container with id 5423a8546089615a9991bfdeab5345de774e5012104e25f4622feec1c58b19dc Oct 30 01:47:07.869944 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:07.869909 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b6b7fc-68dff5d4d-295gs" event={"ID":"a36fc050-879a-4182-ac44-6d3ad2ff189e","Type":"ContainerStarted","Data":"8a81ae121384c2eae6168aa3a7649f9dfc706763a4a454c9c2e956a5d966cd44"} Oct 30 01:47:07.869944 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:07.869960 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b6b7fc-68dff5d4d-295gs" event={"ID":"a36fc050-879a-4182-ac44-6d3ad2ff189e","Type":"ContainerStarted","Data":"5423a8546089615a9991bfdeab5345de774e5012104e25f4622feec1c58b19dc"} Oct 30 01:47:07.870222 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:07.870104 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b6b7fc-68dff5d4d-295gs" Oct 30 01:47:07.871576 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:07.871522 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b6b7fc-546d4d784c-6cjjd" event={"ID":"9061008c-04d0-42d9-96ae-a8ebe16c54b7","Type":"ContainerStarted","Data":"76444066abf01df9d4275cd0ef6c2264589cb0712c1360fe7b6d9aff41ecfc5e"} Oct 30 01:47:07.871719 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:07.871589 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b6b7fc-546d4d784c-6cjjd" event={"ID":"9061008c-04d0-42d9-96ae-a8ebe16c54b7","Type":"ContainerStarted","Data":"46b4ab5bbd837a0303bf3e5882c90314828809b5ad3469b1c57dd27952530154"} Oct 30 01:47:07.871719 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:07.871632 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b6b7fc-546d4d784c-6cjjd" Oct 30 01:47:07.911402 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:07.911340 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b6b7fc-68dff5d4d-295gs" podStartSLOduration=1.911322183 podStartE2EDuration="1.911322183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:47:07.909692017 +0000 UTC m=+1751.487979350" watchObservedRunningTime="2026-10-30 01:47:07.911322183 +0000 UTC m=+1751.489609520" Oct 30 01:47:07.949929 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:07.949866 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b6b7fc-546d4d784c-6cjjd" podStartSLOduration=1.94984679 podStartE2EDuration="1.94984679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:47:07.949119529 +0000 UTC m=+1751.527406865" watchObservedRunningTime="2026-10-30 01:47:07.94984679 +0000 UTC m=+1751.528134128" Oct 30 01:47:08.550395 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:08.550358 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbbfc7ea-d855-4b77-b484-b0b316f40963" path="/var/lib/kubelet/pods/cbbfc7ea-d855-4b77-b484-b0b316f40963/volumes" Oct 30 01:47:09.272056 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:09.272018 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c8d376-76c47844b8-q84n6"] Oct 30 01:47:09.272667 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:09.272651 6221 cpu_manager.go:414] "containerMap: removing container" podUID="cbbfc7ea-d855-4b77-b484-b0b316f40963" containerName="istio-proxy" Oct 30 01:47:09.272721 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:09.272668 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbbfc7ea-d855-4b77-b484-b0b316f40963" containerName="istio-proxy" Oct 30 01:47:09.272773 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:09.272764 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbbfc7ea-d855-4b77-b484-b0b316f40963" containerName="istio-proxy" Oct 30 01:47:09.276233 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:09.276216 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d376-76c47844b8-q84n6" Oct 30 01:47:09.294126 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:09.294100 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c8d376-76c47844b8-q84n6"] Oct 30 01:47:09.303249 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:09.303223 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-c8d376\"" Oct 30 01:47:09.328010 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:09.327978 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d376-f998788b7-gvrjq"] Oct 30 01:47:09.333389 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:09.333359 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d376-f998788b7-gvrjq" Oct 30 01:47:09.348579 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:09.348552 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d376-f998788b7-gvrjq"] Oct 30 01:47:09.422557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:09.407854 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-c8d376-dockercfg-ffrg6\"" Oct 30 01:47:09.465857 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:09.464703 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m75cc\" (UniqueName: \"kubernetes.io/projected/6c21d89b-1302-47e9-a329-ff6e4bd5e10e-kube-api-access-m75cc\") pod \"payload-processing-e2e-shared-b-w0-c8d376-76c47844b8-q84n6\" (UID: \"6c21d89b-1302-47e9-a329-ff6e4bd5e10e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d376-76c47844b8-q84n6" Oct 30 01:47:09.465857 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:09.464855 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c21d89b-1302-47e9-a329-ff6e4bd5e10e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-c8d376-76c47844b8-q84n6\" (UID: \"6c21d89b-1302-47e9-a329-ff6e4bd5e10e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d376-76c47844b8-q84n6" Oct 30 01:47:09.465857 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:09.464908 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t78h2\" (UniqueName: \"kubernetes.io/projected/28dc66b5-3755-4151-a2a3-4dbc793d0c28-kube-api-access-t78h2\") pod \"payload-pre-processing-e2e-shared-b-w0-c8d376-f998788b7-gvrjq\" (UID: \"28dc66b5-3755-4151-a2a3-4dbc793d0c28\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d376-f998788b7-gvrjq" Oct 30 01:47:09.465857 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:09.465007 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28dc66b5-3755-4151-a2a3-4dbc793d0c28-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-c8d376-f998788b7-gvrjq\" (UID: \"28dc66b5-3755-4151-a2a3-4dbc793d0c28\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d376-f998788b7-gvrjq" Oct 30 01:47:09.572050 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:09.572005 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28dc66b5-3755-4151-a2a3-4dbc793d0c28-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-c8d376-f998788b7-gvrjq\" (UID: \"28dc66b5-3755-4151-a2a3-4dbc793d0c28\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d376-f998788b7-gvrjq" Oct 30 01:47:09.572807 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:09.572776 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m75cc\" (UniqueName: \"kubernetes.io/projected/6c21d89b-1302-47e9-a329-ff6e4bd5e10e-kube-api-access-m75cc\") pod \"payload-processing-e2e-shared-b-w0-c8d376-76c47844b8-q84n6\" (UID: \"6c21d89b-1302-47e9-a329-ff6e4bd5e10e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d376-76c47844b8-q84n6" Oct 30 01:47:09.573026 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:09.573006 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c21d89b-1302-47e9-a329-ff6e4bd5e10e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-c8d376-76c47844b8-q84n6\" (UID: \"6c21d89b-1302-47e9-a329-ff6e4bd5e10e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d376-76c47844b8-q84n6" Oct 30 01:47:09.573207 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:09.573190 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t78h2\" (UniqueName: \"kubernetes.io/projected/28dc66b5-3755-4151-a2a3-4dbc793d0c28-kube-api-access-t78h2\") pod \"payload-pre-processing-e2e-shared-b-w0-c8d376-f998788b7-gvrjq\" (UID: \"28dc66b5-3755-4151-a2a3-4dbc793d0c28\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d376-f998788b7-gvrjq" Oct 30 01:47:09.574381 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:09.574354 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28dc66b5-3755-4151-a2a3-4dbc793d0c28-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-c8d376-f998788b7-gvrjq\" (UID: \"28dc66b5-3755-4151-a2a3-4dbc793d0c28\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d376-f998788b7-gvrjq" Oct 30 01:47:09.583313 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:09.583249 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c21d89b-1302-47e9-a329-ff6e4bd5e10e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-c8d376-76c47844b8-q84n6\" (UID: \"6c21d89b-1302-47e9-a329-ff6e4bd5e10e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d376-76c47844b8-q84n6" Oct 30 01:47:09.588243 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:09.588217 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-abb30f-dfb544599-pjbvp" Oct 30 01:47:09.589676 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:09.589655 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-abb30f-549b6b6b86-lsn24" Oct 30 01:47:09.626553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:09.620830 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m75cc\" (UniqueName: \"kubernetes.io/projected/6c21d89b-1302-47e9-a329-ff6e4bd5e10e-kube-api-access-m75cc\") pod \"payload-processing-e2e-shared-b-w0-c8d376-76c47844b8-q84n6\" (UID: \"6c21d89b-1302-47e9-a329-ff6e4bd5e10e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d376-76c47844b8-q84n6" Oct 30 01:47:09.630553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:09.629650 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t78h2\" (UniqueName: \"kubernetes.io/projected/28dc66b5-3755-4151-a2a3-4dbc793d0c28-kube-api-access-t78h2\") pod \"payload-pre-processing-e2e-shared-b-w0-c8d376-f998788b7-gvrjq\" (UID: \"28dc66b5-3755-4151-a2a3-4dbc793d0c28\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d376-f998788b7-gvrjq" Oct 30 01:47:09.655569 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:09.653761 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d376-f998788b7-gvrjq" Oct 30 01:47:09.890409 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:09.890320 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d376-76c47844b8-q84n6" Oct 30 01:47:10.026206 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:10.026176 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d376-f998788b7-gvrjq"] Oct 30 01:47:10.027677 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:47:10.027650 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28dc66b5_3755_4151_a2a3_4dbc793d0c28.slice/crio-dfb23d527f9c4766e5c24889c00889421ebace30fd57964afc3ace91471e2eed WatchSource:0}: Error finding container dfb23d527f9c4766e5c24889c00889421ebace30fd57964afc3ace91471e2eed: Status 404 returned error can't find the container with id dfb23d527f9c4766e5c24889c00889421ebace30fd57964afc3ace91471e2eed Oct 30 01:47:10.190408 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:10.190384 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c8d376-76c47844b8-q84n6"] Oct 30 01:47:10.192863 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:47:10.192834 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c21d89b_1302_47e9_a329_ff6e4bd5e10e.slice/crio-f0e7759918a1245cf1f1f15b4f1bcd099e722428a6a0e4ca46f528922b32fb81 WatchSource:0}: Error finding container f0e7759918a1245cf1f1f15b4f1bcd099e722428a6a0e4ca46f528922b32fb81: Status 404 returned error can't find the container with id f0e7759918a1245cf1f1f15b4f1bcd099e722428a6a0e4ca46f528922b32fb81 Oct 30 01:47:10.891799 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:10.891738 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d376-f998788b7-gvrjq" event={"ID":"28dc66b5-3755-4151-a2a3-4dbc793d0c28","Type":"ContainerStarted","Data":"1e65a93666462732ba6be5470aa969cfdb5fad1a3ec057662f646c46e6680ee6"} Oct 30 01:47:10.891799 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:10.891804 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d376-f998788b7-gvrjq" event={"ID":"28dc66b5-3755-4151-a2a3-4dbc793d0c28","Type":"ContainerStarted","Data":"dfb23d527f9c4766e5c24889c00889421ebace30fd57964afc3ace91471e2eed"} Oct 30 01:47:10.892323 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:10.891932 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d376-f998788b7-gvrjq" Oct 30 01:47:10.893271 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:10.893251 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d376-76c47844b8-q84n6" event={"ID":"6c21d89b-1302-47e9-a329-ff6e4bd5e10e","Type":"ContainerStarted","Data":"a40c22e280ac14b47a74a9c49a6a0a8438e1b46570620095299577cca1f2e3ae"} Oct 30 01:47:10.893379 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:10.893285 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d376-76c47844b8-q84n6" event={"ID":"6c21d89b-1302-47e9-a329-ff6e4bd5e10e","Type":"ContainerStarted","Data":"f0e7759918a1245cf1f1f15b4f1bcd099e722428a6a0e4ca46f528922b32fb81"} Oct 30 01:47:10.893379 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:10.893325 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d376-76c47844b8-q84n6" Oct 30 01:47:10.934557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:10.934486 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d376-f998788b7-gvrjq" podStartSLOduration=1.9344724389999999 podStartE2EDuration="1.934472439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:47:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:47:10.932146132 +0000 UTC m=+1754.510433469" watchObservedRunningTime="2026-10-30 01:47:10.934472439 +0000 UTC m=+1754.512759814" Oct 30 01:47:10.975468 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:10.975419 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d376-76c47844b8-q84n6" podStartSLOduration=1.975404981 podStartE2EDuration="1.975404981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:47:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:47:10.972993711 +0000 UTC m=+1754.551281048" watchObservedRunningTime="2026-10-30 01:47:10.975404981 +0000 UTC m=+1754.553692317" Oct 30 01:47:15.948497 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:15.948459 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2"] Oct 30 01:47:15.955205 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:15.955172 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" Oct 30 01:47:15.974592 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:15.974519 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2"] Oct 30 01:47:15.982400 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:15.982370 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-292018-openshift-default-dockercfg-wh4qz\"" Oct 30 01:47:16.074555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:16.071347 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e229797-d116-4670-86e9-e446396807ad-istio-podinfo\") pod \"e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2\" (UID: \"8e229797-d116-4670-86e9-e446396807ad\") " pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" Oct 30 01:47:16.074555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:16.071423 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e229797-d116-4670-86e9-e446396807ad-istio-token\") pod \"e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2\" (UID: \"8e229797-d116-4670-86e9-e446396807ad\") " pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" Oct 30 01:47:16.074555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:16.071449 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e229797-d116-4670-86e9-e446396807ad-istio-envoy\") pod \"e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2\" (UID: \"8e229797-d116-4670-86e9-e446396807ad\") " pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" Oct 30 01:47:16.074555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:16.071492 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e229797-d116-4670-86e9-e446396807ad-workload-socket\") pod \"e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2\" (UID: \"8e229797-d116-4670-86e9-e446396807ad\") " pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" Oct 30 01:47:16.074555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:16.071548 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e229797-d116-4670-86e9-e446396807ad-credential-socket\") pod \"e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2\" (UID: \"8e229797-d116-4670-86e9-e446396807ad\") " pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" Oct 30 01:47:16.074555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:16.071649 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e229797-d116-4670-86e9-e446396807ad-istio-data\") pod \"e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2\" (UID: \"8e229797-d116-4670-86e9-e446396807ad\") " pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" Oct 30 01:47:16.074555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:16.071734 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e229797-d116-4670-86e9-e446396807ad-workload-certs\") pod \"e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2\" (UID: \"8e229797-d116-4670-86e9-e446396807ad\") " pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" Oct 30 01:47:16.074555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:16.071810 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sblpl\" (UniqueName: \"kubernetes.io/projected/8e229797-d116-4670-86e9-e446396807ad-kube-api-access-sblpl\") pod \"e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2\" (UID: \"8e229797-d116-4670-86e9-e446396807ad\") " pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" Oct 30 01:47:16.074555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:16.071851 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e229797-d116-4670-86e9-e446396807ad-istiod-ca-cert\") pod \"e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2\" (UID: \"8e229797-d116-4670-86e9-e446396807ad\") " pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" Oct 30 01:47:16.179568 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:16.179271 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sblpl\" (UniqueName: \"kubernetes.io/projected/8e229797-d116-4670-86e9-e446396807ad-kube-api-access-sblpl\") pod \"e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2\" (UID: \"8e229797-d116-4670-86e9-e446396807ad\") " pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" Oct 30 01:47:16.179568 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:16.179331 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e229797-d116-4670-86e9-e446396807ad-istiod-ca-cert\") pod \"e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2\" (UID: \"8e229797-d116-4670-86e9-e446396807ad\") " pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" Oct 30 01:47:16.179568 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:16.179415 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e229797-d116-4670-86e9-e446396807ad-istio-podinfo\") pod \"e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2\" (UID: \"8e229797-d116-4670-86e9-e446396807ad\") " pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" Oct 30 01:47:16.179568 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:16.179460 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e229797-d116-4670-86e9-e446396807ad-istio-token\") pod \"e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2\" (UID: \"8e229797-d116-4670-86e9-e446396807ad\") " pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" Oct 30 01:47:16.179568 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:16.179481 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e229797-d116-4670-86e9-e446396807ad-istio-envoy\") pod \"e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2\" (UID: \"8e229797-d116-4670-86e9-e446396807ad\") " pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" Oct 30 01:47:16.179568 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:16.179520 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e229797-d116-4670-86e9-e446396807ad-workload-socket\") pod \"e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2\" (UID: \"8e229797-d116-4670-86e9-e446396807ad\") " pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" Oct 30 01:47:16.179568 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:16.179571 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e229797-d116-4670-86e9-e446396807ad-credential-socket\") pod \"e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2\" (UID: \"8e229797-d116-4670-86e9-e446396807ad\") " pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" Oct 30 01:47:16.180093 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:16.179661 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e229797-d116-4670-86e9-e446396807ad-istio-data\") pod \"e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2\" (UID: \"8e229797-d116-4670-86e9-e446396807ad\") " pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" Oct 30 01:47:16.180093 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:16.179736 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e229797-d116-4670-86e9-e446396807ad-workload-certs\") pod \"e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2\" (UID: \"8e229797-d116-4670-86e9-e446396807ad\") " pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" Oct 30 01:47:16.183552 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:16.180212 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e229797-d116-4670-86e9-e446396807ad-workload-certs\") pod \"e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2\" (UID: \"8e229797-d116-4670-86e9-e446396807ad\") " pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" Oct 30 01:47:16.183552 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:16.181227 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e229797-d116-4670-86e9-e446396807ad-istiod-ca-cert\") pod \"e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2\" (UID: \"8e229797-d116-4670-86e9-e446396807ad\") " pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" Oct 30 01:47:16.183552 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:16.182018 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e229797-d116-4670-86e9-e446396807ad-workload-socket\") pod \"e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2\" (UID: \"8e229797-d116-4670-86e9-e446396807ad\") " pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" Oct 30 01:47:16.183552 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:16.182949 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e229797-d116-4670-86e9-e446396807ad-istio-data\") pod \"e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2\" (UID: \"8e229797-d116-4670-86e9-e446396807ad\") " pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" Oct 30 01:47:16.191567 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:16.189289 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e229797-d116-4670-86e9-e446396807ad-credential-socket\") pod \"e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2\" (UID: \"8e229797-d116-4670-86e9-e446396807ad\") " pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" Oct 30 01:47:16.206881 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:16.203712 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e229797-d116-4670-86e9-e446396807ad-istio-podinfo\") pod \"e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2\" (UID: \"8e229797-d116-4670-86e9-e446396807ad\") " pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" Oct 30 01:47:16.215206 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:16.211876 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e229797-d116-4670-86e9-e446396807ad-istio-envoy\") pod \"e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2\" (UID: \"8e229797-d116-4670-86e9-e446396807ad\") " pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" Oct 30 01:47:16.229103 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:16.228907 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sblpl\" (UniqueName: \"kubernetes.io/projected/8e229797-d116-4670-86e9-e446396807ad-kube-api-access-sblpl\") pod \"e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2\" (UID: \"8e229797-d116-4670-86e9-e446396807ad\") " pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" Oct 30 01:47:16.232556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:16.229498 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e229797-d116-4670-86e9-e446396807ad-istio-token\") pod \"e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2\" (UID: \"8e229797-d116-4670-86e9-e446396807ad\") " pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" Oct 30 01:47:16.287558 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:16.285837 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" Oct 30 01:47:16.646275 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:16.646245 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2"] Oct 30 01:47:16.648369 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:47:16.648327 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e229797_d116_4670_86e9_e446396807ad.slice/crio-ee05c22f08e59944356c7eef459b1c642ee629f65059260fb369f8262a93eace WatchSource:0}: Error finding container ee05c22f08e59944356c7eef459b1c642ee629f65059260fb369f8262a93eace: Status 404 returned error can't find the container with id ee05c22f08e59944356c7eef459b1c642ee629f65059260fb369f8262a93eace Oct 30 01:47:16.651173 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:16.651141 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:47:16.651249 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:16.651208 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:47:16.651249 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:16.651238 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:47:16.939336 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:16.939232 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" event={"ID":"8e229797-d116-4670-86e9-e446396807ad","Type":"ContainerStarted","Data":"4a160237c9824cc92ad28b8c6d3c3961a6c0292b598bbf64354fc22f4219746b"} Oct 30 01:47:16.939519 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:16.939348 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" event={"ID":"8e229797-d116-4670-86e9-e446396807ad","Type":"ContainerStarted","Data":"ee05c22f08e59944356c7eef459b1c642ee629f65059260fb369f8262a93eace"} Oct 30 01:47:17.286580 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:17.286543 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" Oct 30 01:47:17.292715 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:17.292687 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" Oct 30 01:47:17.338232 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:17.338175 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" podStartSLOduration=2.338159085 podStartE2EDuration="2.338159085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:47:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:47:17.002334027 +0000 UTC m=+1760.580621363" watchObservedRunningTime="2026-10-30 01:47:17.338159085 +0000 UTC m=+1760.916446421" Oct 30 01:47:17.945787 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:17.945754 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" Oct 30 01:47:17.946778 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:17.946755 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" Oct 30 01:47:18.881631 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:18.881599 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b6b7fc-546d4d784c-6cjjd" Oct 30 01:47:18.882049 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:18.881678 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b6b7fc-68dff5d4d-295gs" Oct 30 01:47:21.498697 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:47:21.498654 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:47:21.901912 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:21.901877 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d376-76c47844b8-q84n6" Oct 30 01:47:21.902140 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:21.902120 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d376-f998788b7-gvrjq" Oct 30 01:47:24.619522 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:24.619483 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-58632b-78dbfbc54b-857kc"] Oct 30 01:47:24.653003 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:24.652967 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-58632b-fd88c74d7-rmhvd"] Oct 30 01:47:26.924861 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:26.924816 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:47:27.550492 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:27.550451 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d"] Oct 30 01:47:27.551063 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:27.550937 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d" podUID="32d77c5a-d20a-4c87-94a5-bdca98af5135" containerName="istio-proxy" containerID="cri-o://6f1a5ae328f5e5af65f67acb8457b8af4f5930265a59cf780a9af3158a236083" gracePeriod=30 Oct 30 01:47:29.620615 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:29.620558 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-58632b-78dbfbc54b-857kc" podUID="11b32d92-3e3e-48db-a4f5-14b2ed72d7f7" containerName="payload-processing" containerID="cri-o://fa2646bd3c3a283c86cc720f141d0ebef965d17149448aab8a522b749e6281ff" gracePeriod=25 Oct 30 01:47:29.654229 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:29.654186 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-58632b-fd88c74d7-rmhvd" podUID="f5b407f8-baf2-467d-817f-ccfc276df9eb" containerName="payload-pre-processing" containerID="cri-o://68400605f284eefd5461a3e93e8fcb01f9c6e75b9fda6fce08af1bb980ba11d8" gracePeriod=25 Oct 30 01:47:29.935696 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:29.935669 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-58632b-78dbfbc54b-857kc" Oct 30 01:47:29.973959 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:29.973931 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-58632b-fd88c74d7-rmhvd" Oct 30 01:47:30.040398 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:30.040352 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11b32d92-3e3e-48db-a4f5-14b2ed72d7f7-kube-api-access-86npn\" (UniqueName: \"kubernetes.io/projected/11b32d92-3e3e-48db-a4f5-14b2ed72d7f7-kube-api-access-86npn\") pod \"11b32d92-3e3e-48db-a4f5-14b2ed72d7f7\" (UID: \"11b32d92-3e3e-48db-a4f5-14b2ed72d7f7\") " Oct 30 01:47:30.040649 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:30.040525 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11b32d92-3e3e-48db-a4f5-14b2ed72d7f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11b32d92-3e3e-48db-a4f5-14b2ed72d7f7-plugins-config-volume\") pod \"11b32d92-3e3e-48db-a4f5-14b2ed72d7f7\" (UID: \"11b32d92-3e3e-48db-a4f5-14b2ed72d7f7\") " Oct 30 01:47:30.040898 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:30.040875 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11b32d92-3e3e-48db-a4f5-14b2ed72d7f7-plugins-config-volume" pod "11b32d92-3e3e-48db-a4f5-14b2ed72d7f7" (UID: "11b32d92-3e3e-48db-a4f5-14b2ed72d7f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:47:30.041380 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:30.041349 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11b32d92-3e3e-48db-a4f5-14b2ed72d7f7-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:47:30.042867 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:30.042843 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11b32d92-3e3e-48db-a4f5-14b2ed72d7f7-kube-api-access-86npn" pod "11b32d92-3e3e-48db-a4f5-14b2ed72d7f7" (UID: "11b32d92-3e3e-48db-a4f5-14b2ed72d7f7"). InnerVolumeSpecName "kube-api-access-86npn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:47:30.132881 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:30.132787 6221 generic.go:356] "Generic (PLEG): container finished" podID="11b32d92-3e3e-48db-a4f5-14b2ed72d7f7" containerID="fa2646bd3c3a283c86cc720f141d0ebef965d17149448aab8a522b749e6281ff" exitCode=0 Oct 30 01:47:30.132881 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:30.132836 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-58632b-78dbfbc54b-857kc" Oct 30 01:47:30.133102 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:30.132879 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-58632b-78dbfbc54b-857kc" event={"ID":"11b32d92-3e3e-48db-a4f5-14b2ed72d7f7","Type":"ContainerDied","Data":"fa2646bd3c3a283c86cc720f141d0ebef965d17149448aab8a522b749e6281ff"} Oct 30 01:47:30.133102 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:30.132974 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-58632b-78dbfbc54b-857kc" event={"ID":"11b32d92-3e3e-48db-a4f5-14b2ed72d7f7","Type":"ContainerDied","Data":"0e2c0edaef9734d1f2166645ae1567f8a97cbd2510a296c2496f2967f4fbe423"} Oct 30 01:47:30.133102 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:30.133010 6221 scope.go:122] "RemoveContainer" containerID="fa2646bd3c3a283c86cc720f141d0ebef965d17149448aab8a522b749e6281ff" Oct 30 01:47:30.134466 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:30.134441 6221 generic.go:356] "Generic (PLEG): container finished" podID="f5b407f8-baf2-467d-817f-ccfc276df9eb" containerID="68400605f284eefd5461a3e93e8fcb01f9c6e75b9fda6fce08af1bb980ba11d8" exitCode=0 Oct 30 01:47:30.134623 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:30.134487 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-58632b-fd88c74d7-rmhvd" event={"ID":"f5b407f8-baf2-467d-817f-ccfc276df9eb","Type":"ContainerDied","Data":"68400605f284eefd5461a3e93e8fcb01f9c6e75b9fda6fce08af1bb980ba11d8"} Oct 30 01:47:30.134623 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:30.134582 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-58632b-fd88c74d7-rmhvd" event={"ID":"f5b407f8-baf2-467d-817f-ccfc276df9eb","Type":"ContainerDied","Data":"0ffa03c0a287f3d50ca88212e0cd92a69747e701d2bed2f08c9c3520ad4d1fed"} Oct 30 01:47:30.134623 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:30.134493 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-58632b-fd88c74d7-rmhvd" Oct 30 01:47:30.142022 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:30.141992 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5b407f8-baf2-467d-817f-ccfc276df9eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5b407f8-baf2-467d-817f-ccfc276df9eb-plugins-config-volume\") pod \"f5b407f8-baf2-467d-817f-ccfc276df9eb\" (UID: \"f5b407f8-baf2-467d-817f-ccfc276df9eb\") " Oct 30 01:47:30.142179 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:30.142158 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5b407f8-baf2-467d-817f-ccfc276df9eb-kube-api-access-jw54m\" (UniqueName: \"kubernetes.io/projected/f5b407f8-baf2-467d-817f-ccfc276df9eb-kube-api-access-jw54m\") pod \"f5b407f8-baf2-467d-817f-ccfc276df9eb\" (UID: \"f5b407f8-baf2-467d-817f-ccfc276df9eb\") " Oct 30 01:47:30.142519 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:30.142493 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5b407f8-baf2-467d-817f-ccfc276df9eb-plugins-config-volume" pod "f5b407f8-baf2-467d-817f-ccfc276df9eb" (UID: "f5b407f8-baf2-467d-817f-ccfc276df9eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:47:30.142749 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:30.142731 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5b407f8-baf2-467d-817f-ccfc276df9eb-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:47:30.142822 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:30.142753 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86npn\" (UniqueName: \"kubernetes.io/projected/11b32d92-3e3e-48db-a4f5-14b2ed72d7f7-kube-api-access-86npn\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:47:30.144215 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:30.144188 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5b407f8-baf2-467d-817f-ccfc276df9eb-kube-api-access-jw54m" pod "f5b407f8-baf2-467d-817f-ccfc276df9eb" (UID: "f5b407f8-baf2-467d-817f-ccfc276df9eb"). InnerVolumeSpecName "kube-api-access-jw54m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:47:30.144458 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:30.144436 6221 scope.go:122] "RemoveContainer" containerID="fa2646bd3c3a283c86cc720f141d0ebef965d17149448aab8a522b749e6281ff" Oct 30 01:47:30.144787 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:47:30.144761 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa2646bd3c3a283c86cc720f141d0ebef965d17149448aab8a522b749e6281ff\": container with ID starting with fa2646bd3c3a283c86cc720f141d0ebef965d17149448aab8a522b749e6281ff not found: ID does not exist" containerID="fa2646bd3c3a283c86cc720f141d0ebef965d17149448aab8a522b749e6281ff" Oct 30 01:47:30.144863 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:30.144796 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa2646bd3c3a283c86cc720f141d0ebef965d17149448aab8a522b749e6281ff"} err="failed to get container status \"fa2646bd3c3a283c86cc720f141d0ebef965d17149448aab8a522b749e6281ff\": rpc error: code = NotFound desc = could not find container \"fa2646bd3c3a283c86cc720f141d0ebef965d17149448aab8a522b749e6281ff\": container with ID starting with fa2646bd3c3a283c86cc720f141d0ebef965d17149448aab8a522b749e6281ff not found: ID does not exist" Oct 30 01:47:30.144863 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:30.144814 6221 scope.go:122] "RemoveContainer" containerID="68400605f284eefd5461a3e93e8fcb01f9c6e75b9fda6fce08af1bb980ba11d8" Oct 30 01:47:30.154281 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:30.154258 6221 scope.go:122] "RemoveContainer" containerID="68400605f284eefd5461a3e93e8fcb01f9c6e75b9fda6fce08af1bb980ba11d8" Oct 30 01:47:30.154665 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:47:30.154644 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68400605f284eefd5461a3e93e8fcb01f9c6e75b9fda6fce08af1bb980ba11d8\": container with ID starting with 68400605f284eefd5461a3e93e8fcb01f9c6e75b9fda6fce08af1bb980ba11d8 not found: ID does not exist" containerID="68400605f284eefd5461a3e93e8fcb01f9c6e75b9fda6fce08af1bb980ba11d8" Oct 30 01:47:30.154721 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:30.154674 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68400605f284eefd5461a3e93e8fcb01f9c6e75b9fda6fce08af1bb980ba11d8"} err="failed to get container status \"68400605f284eefd5461a3e93e8fcb01f9c6e75b9fda6fce08af1bb980ba11d8\": rpc error: code = NotFound desc = could not find container \"68400605f284eefd5461a3e93e8fcb01f9c6e75b9fda6fce08af1bb980ba11d8\": container with ID starting with 68400605f284eefd5461a3e93e8fcb01f9c6e75b9fda6fce08af1bb980ba11d8 not found: ID does not exist" Oct 30 01:47:30.188561 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:30.188511 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-58632b-78dbfbc54b-857kc"] Oct 30 01:47:30.191807 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:30.191779 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-58632b-78dbfbc54b-857kc"] Oct 30 01:47:30.244480 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:30.244437 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw54m\" (UniqueName: \"kubernetes.io/projected/f5b407f8-baf2-467d-817f-ccfc276df9eb-kube-api-access-jw54m\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:47:30.489156 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:30.489111 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-58632b-fd88c74d7-rmhvd"] Oct 30 01:47:30.493324 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:30.493295 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-58632b-fd88c74d7-rmhvd"] Oct 30 01:47:30.548367 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:30.548324 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11b32d92-3e3e-48db-a4f5-14b2ed72d7f7" path="/var/lib/kubelet/pods/11b32d92-3e3e-48db-a4f5-14b2ed72d7f7/volumes" Oct 30 01:47:30.548861 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:30.548843 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5b407f8-baf2-467d-817f-ccfc276df9eb" path="/var/lib/kubelet/pods/f5b407f8-baf2-467d-817f-ccfc276df9eb/volumes" Oct 30 01:47:32.859170 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:32.859144 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d" Oct 30 01:47:32.983622 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:32.983584 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32d77c5a-d20a-4c87-94a5-bdca98af5135-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32d77c5a-d20a-4c87-94a5-bdca98af5135-istiod-ca-cert\") pod \"32d77c5a-d20a-4c87-94a5-bdca98af5135\" (UID: \"32d77c5a-d20a-4c87-94a5-bdca98af5135\") " Oct 30 01:47:32.983836 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:32.983666 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32d77c5a-d20a-4c87-94a5-bdca98af5135-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32d77c5a-d20a-4c87-94a5-bdca98af5135-workload-socket\") pod \"32d77c5a-d20a-4c87-94a5-bdca98af5135\" (UID: \"32d77c5a-d20a-4c87-94a5-bdca98af5135\") " Oct 30 01:47:32.983836 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:32.983709 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32d77c5a-d20a-4c87-94a5-bdca98af5135-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32d77c5a-d20a-4c87-94a5-bdca98af5135-istio-envoy\") pod \"32d77c5a-d20a-4c87-94a5-bdca98af5135\" (UID: \"32d77c5a-d20a-4c87-94a5-bdca98af5135\") " Oct 30 01:47:32.983836 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:32.983747 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32d77c5a-d20a-4c87-94a5-bdca98af5135-istio-token\" (UniqueName: \"kubernetes.io/projected/32d77c5a-d20a-4c87-94a5-bdca98af5135-istio-token\") pod \"32d77c5a-d20a-4c87-94a5-bdca98af5135\" (UID: \"32d77c5a-d20a-4c87-94a5-bdca98af5135\") " Oct 30 01:47:32.983836 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:32.983793 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32d77c5a-d20a-4c87-94a5-bdca98af5135-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32d77c5a-d20a-4c87-94a5-bdca98af5135-credential-socket\") pod \"32d77c5a-d20a-4c87-94a5-bdca98af5135\" (UID: \"32d77c5a-d20a-4c87-94a5-bdca98af5135\") " Oct 30 01:47:32.983836 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:32.983823 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32d77c5a-d20a-4c87-94a5-bdca98af5135-kube-api-access-lrrpr\" (UniqueName: \"kubernetes.io/projected/32d77c5a-d20a-4c87-94a5-bdca98af5135-kube-api-access-lrrpr\") pod \"32d77c5a-d20a-4c87-94a5-bdca98af5135\" (UID: \"32d77c5a-d20a-4c87-94a5-bdca98af5135\") " Oct 30 01:47:32.984112 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:32.983874 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32d77c5a-d20a-4c87-94a5-bdca98af5135-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32d77c5a-d20a-4c87-94a5-bdca98af5135-istio-podinfo\") pod \"32d77c5a-d20a-4c87-94a5-bdca98af5135\" (UID: \"32d77c5a-d20a-4c87-94a5-bdca98af5135\") " Oct 30 01:47:32.984112 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:32.983920 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32d77c5a-d20a-4c87-94a5-bdca98af5135-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32d77c5a-d20a-4c87-94a5-bdca98af5135-istio-data\") pod \"32d77c5a-d20a-4c87-94a5-bdca98af5135\" (UID: \"32d77c5a-d20a-4c87-94a5-bdca98af5135\") " Oct 30 01:47:32.984112 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:32.983976 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32d77c5a-d20a-4c87-94a5-bdca98af5135-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32d77c5a-d20a-4c87-94a5-bdca98af5135-workload-certs\") pod \"32d77c5a-d20a-4c87-94a5-bdca98af5135\" (UID: \"32d77c5a-d20a-4c87-94a5-bdca98af5135\") " Oct 30 01:47:32.984112 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:32.984020 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32d77c5a-d20a-4c87-94a5-bdca98af5135-workload-socket" pod "32d77c5a-d20a-4c87-94a5-bdca98af5135" (UID: "32d77c5a-d20a-4c87-94a5-bdca98af5135"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:47:32.984320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:32.984169 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32d77c5a-d20a-4c87-94a5-bdca98af5135-credential-socket" pod "32d77c5a-d20a-4c87-94a5-bdca98af5135" (UID: "32d77c5a-d20a-4c87-94a5-bdca98af5135"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:47:32.984320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:32.984203 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32d77c5a-d20a-4c87-94a5-bdca98af5135-istiod-ca-cert" pod "32d77c5a-d20a-4c87-94a5-bdca98af5135" (UID: "32d77c5a-d20a-4c87-94a5-bdca98af5135"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:47:32.984524 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:32.984406 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32d77c5a-d20a-4c87-94a5-bdca98af5135-workload-certs" pod "32d77c5a-d20a-4c87-94a5-bdca98af5135" (UID: "32d77c5a-d20a-4c87-94a5-bdca98af5135"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:47:32.984524 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:32.984421 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32d77c5a-d20a-4c87-94a5-bdca98af5135-istio-data" pod "32d77c5a-d20a-4c87-94a5-bdca98af5135" (UID: "32d77c5a-d20a-4c87-94a5-bdca98af5135"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:47:32.984981 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:32.984876 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32d77c5a-d20a-4c87-94a5-bdca98af5135-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:47:32.984981 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:32.984904 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32d77c5a-d20a-4c87-94a5-bdca98af5135-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:47:32.984981 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:32.984919 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32d77c5a-d20a-4c87-94a5-bdca98af5135-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:47:32.984981 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:32.984935 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32d77c5a-d20a-4c87-94a5-bdca98af5135-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:47:32.984981 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:32.984952 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32d77c5a-d20a-4c87-94a5-bdca98af5135-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:47:32.986254 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:32.986233 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32d77c5a-d20a-4c87-94a5-bdca98af5135-istio-envoy" pod "32d77c5a-d20a-4c87-94a5-bdca98af5135" (UID: "32d77c5a-d20a-4c87-94a5-bdca98af5135"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:47:32.986382 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:32.986357 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32d77c5a-d20a-4c87-94a5-bdca98af5135-kube-api-access-lrrpr" pod "32d77c5a-d20a-4c87-94a5-bdca98af5135" (UID: "32d77c5a-d20a-4c87-94a5-bdca98af5135"). InnerVolumeSpecName "kube-api-access-lrrpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:47:32.986517 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:32.986499 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32d77c5a-d20a-4c87-94a5-bdca98af5135-istio-token" pod "32d77c5a-d20a-4c87-94a5-bdca98af5135" (UID: "32d77c5a-d20a-4c87-94a5-bdca98af5135"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:47:32.986597 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:32.986581 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32d77c5a-d20a-4c87-94a5-bdca98af5135-istio-podinfo" pod "32d77c5a-d20a-4c87-94a5-bdca98af5135" (UID: "32d77c5a-d20a-4c87-94a5-bdca98af5135"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:47:33.086173 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:33.086086 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32d77c5a-d20a-4c87-94a5-bdca98af5135-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:47:33.086173 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:33.086117 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32d77c5a-d20a-4c87-94a5-bdca98af5135-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:47:33.086173 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:33.086127 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrrpr\" (UniqueName: \"kubernetes.io/projected/32d77c5a-d20a-4c87-94a5-bdca98af5135-kube-api-access-lrrpr\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:47:33.086173 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:33.086137 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32d77c5a-d20a-4c87-94a5-bdca98af5135-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:47:33.155856 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:33.155815 6221 generic.go:356] "Generic (PLEG): container finished" podID="32d77c5a-d20a-4c87-94a5-bdca98af5135" containerID="6f1a5ae328f5e5af65f67acb8457b8af4f5930265a59cf780a9af3158a236083" exitCode=0 Oct 30 01:47:33.156048 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:33.155874 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d" Oct 30 01:47:33.156048 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:33.155899 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d" event={"ID":"32d77c5a-d20a-4c87-94a5-bdca98af5135","Type":"ContainerDied","Data":"6f1a5ae328f5e5af65f67acb8457b8af4f5930265a59cf780a9af3158a236083"} Oct 30 01:47:33.156048 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:33.155965 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d" event={"ID":"32d77c5a-d20a-4c87-94a5-bdca98af5135","Type":"ContainerDied","Data":"9c8f86e7c15c56f5789cf6d24b84033cba1c37f6817765370b6a28a724f3619b"} Oct 30 01:47:33.156048 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:33.155996 6221 scope.go:122] "RemoveContainer" containerID="6f1a5ae328f5e5af65f67acb8457b8af4f5930265a59cf780a9af3158a236083" Oct 30 01:47:33.166974 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:33.166952 6221 scope.go:122] "RemoveContainer" containerID="6f1a5ae328f5e5af65f67acb8457b8af4f5930265a59cf780a9af3158a236083" Oct 30 01:47:33.167260 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:47:33.167240 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f1a5ae328f5e5af65f67acb8457b8af4f5930265a59cf780a9af3158a236083\": container with ID starting with 6f1a5ae328f5e5af65f67acb8457b8af4f5930265a59cf780a9af3158a236083 not found: ID does not exist" containerID="6f1a5ae328f5e5af65f67acb8457b8af4f5930265a59cf780a9af3158a236083" Oct 30 01:47:33.167309 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:33.167269 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f1a5ae328f5e5af65f67acb8457b8af4f5930265a59cf780a9af3158a236083"} err="failed to get container status \"6f1a5ae328f5e5af65f67acb8457b8af4f5930265a59cf780a9af3158a236083\": rpc error: code = NotFound desc = could not find container \"6f1a5ae328f5e5af65f67acb8457b8af4f5930265a59cf780a9af3158a236083\": container with ID starting with 6f1a5ae328f5e5af65f67acb8457b8af4f5930265a59cf780a9af3158a236083 not found: ID does not exist" Oct 30 01:47:33.211214 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:33.211176 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d"] Oct 30 01:47:33.215495 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:33.215468 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jsg6d"] Oct 30 01:47:34.548056 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:34.548008 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32d77c5a-d20a-4c87-94a5-bdca98af5135" path="/var/lib/kubelet/pods/32d77c5a-d20a-4c87-94a5-bdca98af5135/volumes" Oct 30 01:47:36.896221 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:36.896186 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp"] Oct 30 01:47:36.896817 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:36.896803 6221 cpu_manager.go:414] "containerMap: removing container" podUID="11b32d92-3e3e-48db-a4f5-14b2ed72d7f7" containerName="payload-processing" Oct 30 01:47:36.896865 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:36.896818 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11b32d92-3e3e-48db-a4f5-14b2ed72d7f7" containerName="payload-processing" Oct 30 01:47:36.896865 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:36.896829 6221 cpu_manager.go:414] "containerMap: removing container" podUID="f5b407f8-baf2-467d-817f-ccfc276df9eb" containerName="payload-pre-processing" Oct 30 01:47:36.896865 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:36.896834 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5b407f8-baf2-467d-817f-ccfc276df9eb" containerName="payload-pre-processing" Oct 30 01:47:36.896865 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:36.896862 6221 cpu_manager.go:414] "containerMap: removing container" podUID="32d77c5a-d20a-4c87-94a5-bdca98af5135" containerName="istio-proxy" Oct 30 01:47:36.896990 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:36.896868 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32d77c5a-d20a-4c87-94a5-bdca98af5135" containerName="istio-proxy" Oct 30 01:47:36.896990 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:36.896963 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="11b32d92-3e3e-48db-a4f5-14b2ed72d7f7" containerName="payload-processing" Oct 30 01:47:36.896990 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:36.896972 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5b407f8-baf2-467d-817f-ccfc276df9eb" containerName="payload-pre-processing" Oct 30 01:47:36.896990 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:36.896983 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="32d77c5a-d20a-4c87-94a5-bdca98af5135" containerName="istio-proxy" Oct 30 01:47:36.902093 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:36.902064 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" Oct 30 01:47:36.920457 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:36.920430 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-861cd2-openshift-default-dockercfg-ljqsf\"" Oct 30 01:47:36.938227 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:36.938189 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp"] Oct 30 01:47:36.949422 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:36.949378 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjtkj\" (UniqueName: \"kubernetes.io/projected/51dc5d9e-31aa-4246-9bd6-8953c55424f1-kube-api-access-zjtkj\") pod \"e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp\" (UID: \"51dc5d9e-31aa-4246-9bd6-8953c55424f1\") " pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" Oct 30 01:47:36.949656 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:36.949441 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51dc5d9e-31aa-4246-9bd6-8953c55424f1-istio-data\") pod \"e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp\" (UID: \"51dc5d9e-31aa-4246-9bd6-8953c55424f1\") " pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" Oct 30 01:47:36.949656 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:36.949566 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51dc5d9e-31aa-4246-9bd6-8953c55424f1-workload-certs\") pod \"e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp\" (UID: \"51dc5d9e-31aa-4246-9bd6-8953c55424f1\") " pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" Oct 30 01:47:36.949656 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:36.949602 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51dc5d9e-31aa-4246-9bd6-8953c55424f1-istio-token\") pod \"e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp\" (UID: \"51dc5d9e-31aa-4246-9bd6-8953c55424f1\") " pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" Oct 30 01:47:36.949846 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:36.949690 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51dc5d9e-31aa-4246-9bd6-8953c55424f1-istio-podinfo\") pod \"e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp\" (UID: \"51dc5d9e-31aa-4246-9bd6-8953c55424f1\") " pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" Oct 30 01:47:36.949846 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:36.949753 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51dc5d9e-31aa-4246-9bd6-8953c55424f1-istiod-ca-cert\") pod \"e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp\" (UID: \"51dc5d9e-31aa-4246-9bd6-8953c55424f1\") " pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" Oct 30 01:47:36.949950 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:36.949924 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51dc5d9e-31aa-4246-9bd6-8953c55424f1-credential-socket\") pod \"e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp\" (UID: \"51dc5d9e-31aa-4246-9bd6-8953c55424f1\") " pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" Oct 30 01:47:36.950028 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:36.950008 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51dc5d9e-31aa-4246-9bd6-8953c55424f1-istio-envoy\") pod \"e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp\" (UID: \"51dc5d9e-31aa-4246-9bd6-8953c55424f1\") " pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" Oct 30 01:47:36.950133 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:36.950114 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51dc5d9e-31aa-4246-9bd6-8953c55424f1-workload-socket\") pod \"e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp\" (UID: \"51dc5d9e-31aa-4246-9bd6-8953c55424f1\") " pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" Oct 30 01:47:37.066557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:37.065548 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51dc5d9e-31aa-4246-9bd6-8953c55424f1-istio-envoy\") pod \"e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp\" (UID: \"51dc5d9e-31aa-4246-9bd6-8953c55424f1\") " pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" Oct 30 01:47:37.066557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:37.065656 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51dc5d9e-31aa-4246-9bd6-8953c55424f1-workload-socket\") pod \"e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp\" (UID: \"51dc5d9e-31aa-4246-9bd6-8953c55424f1\") " pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" Oct 30 01:47:37.066557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:37.065701 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjtkj\" (UniqueName: \"kubernetes.io/projected/51dc5d9e-31aa-4246-9bd6-8953c55424f1-kube-api-access-zjtkj\") pod \"e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp\" (UID: \"51dc5d9e-31aa-4246-9bd6-8953c55424f1\") " pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" Oct 30 01:47:37.066557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:37.065737 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51dc5d9e-31aa-4246-9bd6-8953c55424f1-istio-data\") pod \"e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp\" (UID: \"51dc5d9e-31aa-4246-9bd6-8953c55424f1\") " pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" Oct 30 01:47:37.066557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:37.065777 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51dc5d9e-31aa-4246-9bd6-8953c55424f1-workload-certs\") pod \"e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp\" (UID: \"51dc5d9e-31aa-4246-9bd6-8953c55424f1\") " pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" Oct 30 01:47:37.066557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:37.065799 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51dc5d9e-31aa-4246-9bd6-8953c55424f1-istio-token\") pod \"e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp\" (UID: \"51dc5d9e-31aa-4246-9bd6-8953c55424f1\") " pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" Oct 30 01:47:37.066557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:37.065855 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51dc5d9e-31aa-4246-9bd6-8953c55424f1-istio-podinfo\") pod \"e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp\" (UID: \"51dc5d9e-31aa-4246-9bd6-8953c55424f1\") " pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" Oct 30 01:47:37.066557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:37.065891 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51dc5d9e-31aa-4246-9bd6-8953c55424f1-istiod-ca-cert\") pod \"e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp\" (UID: \"51dc5d9e-31aa-4246-9bd6-8953c55424f1\") " pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" Oct 30 01:47:37.066557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:37.066005 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51dc5d9e-31aa-4246-9bd6-8953c55424f1-credential-socket\") pod \"e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp\" (UID: \"51dc5d9e-31aa-4246-9bd6-8953c55424f1\") " pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" Oct 30 01:47:37.066557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:37.066474 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51dc5d9e-31aa-4246-9bd6-8953c55424f1-credential-socket\") pod \"e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp\" (UID: \"51dc5d9e-31aa-4246-9bd6-8953c55424f1\") " pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" Oct 30 01:47:37.084552 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:37.083994 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51dc5d9e-31aa-4246-9bd6-8953c55424f1-workload-certs\") pod \"e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp\" (UID: \"51dc5d9e-31aa-4246-9bd6-8953c55424f1\") " pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" Oct 30 01:47:37.084552 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:37.084236 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51dc5d9e-31aa-4246-9bd6-8953c55424f1-workload-socket\") pod \"e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp\" (UID: \"51dc5d9e-31aa-4246-9bd6-8953c55424f1\") " pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" Oct 30 01:47:37.084826 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:37.084795 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51dc5d9e-31aa-4246-9bd6-8953c55424f1-istio-data\") pod \"e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp\" (UID: \"51dc5d9e-31aa-4246-9bd6-8953c55424f1\") " pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" Oct 30 01:47:37.111556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:37.102230 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51dc5d9e-31aa-4246-9bd6-8953c55424f1-istiod-ca-cert\") pod \"e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp\" (UID: \"51dc5d9e-31aa-4246-9bd6-8953c55424f1\") " pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" Oct 30 01:47:37.118554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:37.116897 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51dc5d9e-31aa-4246-9bd6-8953c55424f1-istio-envoy\") pod \"e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp\" (UID: \"51dc5d9e-31aa-4246-9bd6-8953c55424f1\") " pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" Oct 30 01:47:37.118554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:37.117705 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51dc5d9e-31aa-4246-9bd6-8953c55424f1-istio-podinfo\") pod \"e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp\" (UID: \"51dc5d9e-31aa-4246-9bd6-8953c55424f1\") " pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" Oct 30 01:47:37.118554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:37.118394 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjtkj\" (UniqueName: \"kubernetes.io/projected/51dc5d9e-31aa-4246-9bd6-8953c55424f1-kube-api-access-zjtkj\") pod \"e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp\" (UID: \"51dc5d9e-31aa-4246-9bd6-8953c55424f1\") " pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" Oct 30 01:47:37.146598 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:37.145035 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51dc5d9e-31aa-4246-9bd6-8953c55424f1-istio-token\") pod \"e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp\" (UID: \"51dc5d9e-31aa-4246-9bd6-8953c55424f1\") " pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" Oct 30 01:47:37.220578 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:37.219804 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" Oct 30 01:47:37.546094 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:37.546065 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp"] Oct 30 01:47:37.548637 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:47:37.548609 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51dc5d9e_31aa_4246_9bd6_8953c55424f1.slice/crio-3ad2eaee5e00a4d627b321483b69aa83d4b4c2b1f734a5757709fd16e43e624a WatchSource:0}: Error finding container 3ad2eaee5e00a4d627b321483b69aa83d4b4c2b1f734a5757709fd16e43e624a: Status 404 returned error can't find the container with id 3ad2eaee5e00a4d627b321483b69aa83d4b4c2b1f734a5757709fd16e43e624a Oct 30 01:47:37.551126 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:37.551094 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:47:37.551187 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:37.551175 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:47:37.551229 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:37.551206 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:47:38.198969 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:38.198934 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" event={"ID":"51dc5d9e-31aa-4246-9bd6-8953c55424f1","Type":"ContainerStarted","Data":"43303036d7d0a45bfc51c6bd52645cd1b9c19a912ac61ad1fe416e6f4e6d3baa"} Oct 30 01:47:38.199407 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:38.198987 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" event={"ID":"51dc5d9e-31aa-4246-9bd6-8953c55424f1","Type":"ContainerStarted","Data":"3ad2eaee5e00a4d627b321483b69aa83d4b4c2b1f734a5757709fd16e43e624a"} Oct 30 01:47:38.221168 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:38.221131 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" Oct 30 01:47:38.227181 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:38.227153 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" Oct 30 01:47:38.268210 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:38.268141 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" podStartSLOduration=2.268126972 podStartE2EDuration="2.268126972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:47:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:47:38.255300058 +0000 UTC m=+1781.833587394" watchObservedRunningTime="2026-10-30 01:47:38.268126972 +0000 UTC m=+1781.846414307" Oct 30 01:47:39.206197 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:39.206165 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" Oct 30 01:47:39.207317 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:39.207296 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" Oct 30 01:47:42.732564 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:42.731745 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-861cd2-6477f778c6-8cbg5"] Oct 30 01:47:42.793643 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:42.788353 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-861cd2-6477f778c6-8cbg5" Oct 30 01:47:42.820568 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:42.808160 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-861cd2-dockercfg-vqmqv\"" Oct 30 01:47:42.820568 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:42.818708 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-861cd2\"" Oct 30 01:47:42.824611 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:42.823705 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-861cd2-6477f778c6-8cbg5"] Oct 30 01:47:42.923500 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:42.923460 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxrkx\" (UniqueName: \"kubernetes.io/projected/e4160a65-33dc-4c24-8a6d-f24c37cb3ebe-kube-api-access-hxrkx\") pod \"payload-processing-e2e-shared-b-w1-861cd2-6477f778c6-8cbg5\" (UID: \"e4160a65-33dc-4c24-8a6d-f24c37cb3ebe\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-861cd2-6477f778c6-8cbg5" Oct 30 01:47:42.923711 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:42.923687 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4160a65-33dc-4c24-8a6d-f24c37cb3ebe-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-861cd2-6477f778c6-8cbg5\" (UID: \"e4160a65-33dc-4c24-8a6d-f24c37cb3ebe\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-861cd2-6477f778c6-8cbg5" Oct 30 01:47:43.025023 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:43.024923 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxrkx\" (UniqueName: \"kubernetes.io/projected/e4160a65-33dc-4c24-8a6d-f24c37cb3ebe-kube-api-access-hxrkx\") pod \"payload-processing-e2e-shared-b-w1-861cd2-6477f778c6-8cbg5\" (UID: \"e4160a65-33dc-4c24-8a6d-f24c37cb3ebe\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-861cd2-6477f778c6-8cbg5" Oct 30 01:47:43.025222 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:43.025053 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4160a65-33dc-4c24-8a6d-f24c37cb3ebe-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-861cd2-6477f778c6-8cbg5\" (UID: \"e4160a65-33dc-4c24-8a6d-f24c37cb3ebe\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-861cd2-6477f778c6-8cbg5" Oct 30 01:47:43.026041 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:43.026014 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4160a65-33dc-4c24-8a6d-f24c37cb3ebe-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-861cd2-6477f778c6-8cbg5\" (UID: \"e4160a65-33dc-4c24-8a6d-f24c37cb3ebe\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-861cd2-6477f778c6-8cbg5" Oct 30 01:47:43.044449 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:43.044412 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxrkx\" (UniqueName: \"kubernetes.io/projected/e4160a65-33dc-4c24-8a6d-f24c37cb3ebe-kube-api-access-hxrkx\") pod \"payload-processing-e2e-shared-b-w1-861cd2-6477f778c6-8cbg5\" (UID: \"e4160a65-33dc-4c24-8a6d-f24c37cb3ebe\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-861cd2-6477f778c6-8cbg5" Oct 30 01:47:43.216424 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:43.216369 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-861cd2-6477f778c6-8cbg5" Oct 30 01:47:43.583046 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:43.583022 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-861cd2-6477f778c6-8cbg5"] Oct 30 01:47:43.585933 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:47:43.585901 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4160a65_33dc_4c24_8a6d_f24c37cb3ebe.slice/crio-d579666b42848b87f5c6456b9ec2d38265c1e6a7886d3ef59113763e62ab80da WatchSource:0}: Error finding container d579666b42848b87f5c6456b9ec2d38265c1e6a7886d3ef59113763e62ab80da: Status 404 returned error can't find the container with id d579666b42848b87f5c6456b9ec2d38265c1e6a7886d3ef59113763e62ab80da Oct 30 01:47:44.299604 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:44.299560 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-861cd2-6477f778c6-8cbg5" event={"ID":"e4160a65-33dc-4c24-8a6d-f24c37cb3ebe","Type":"ContainerStarted","Data":"78f3e7ed3ccaf97a4e8ee4df3cb62753455c143dd03264f121ea112d0763ce42"} Oct 30 01:47:44.300066 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:44.299623 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-861cd2-6477f778c6-8cbg5" event={"ID":"e4160a65-33dc-4c24-8a6d-f24c37cb3ebe","Type":"ContainerStarted","Data":"d579666b42848b87f5c6456b9ec2d38265c1e6a7886d3ef59113763e62ab80da"} Oct 30 01:47:44.300066 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:44.299775 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-861cd2-6477f778c6-8cbg5" Oct 30 01:47:44.341053 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:44.340995 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-861cd2-6477f778c6-8cbg5" podStartSLOduration=2.340979898 podStartE2EDuration="2.340979898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:47:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:47:44.337416665 +0000 UTC m=+1787.915704002" watchObservedRunningTime="2026-10-30 01:47:44.340979898 +0000 UTC m=+1787.919267260" Oct 30 01:47:46.715023 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:46.714986 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-58632b-58b9fff455-r2t2s"] Oct 30 01:47:46.715944 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:46.715905 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-58632b-58b9fff455-r2t2s" podUID="f125248b-5ef7-4283-bb8c-44549060fbb8" containerName="maas-api" containerID="cri-o://cbf64338b43d6bf513539febcc6632d13138b9f1170bd1f062f8244166c28575" gracePeriod=30 Oct 30 01:47:47.066143 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:47.066107 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-58632b-58b9fff455-r2t2s" Oct 30 01:47:47.089740 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:47.089683 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f125248b-5ef7-4283-bb8c-44549060fbb8-kube-api-access-k4lxn\" (UniqueName: \"kubernetes.io/projected/f125248b-5ef7-4283-bb8c-44549060fbb8-kube-api-access-k4lxn\") pod \"f125248b-5ef7-4283-bb8c-44549060fbb8\" (UID: \"f125248b-5ef7-4283-bb8c-44549060fbb8\") " Oct 30 01:47:47.089963 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:47.089770 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f125248b-5ef7-4283-bb8c-44549060fbb8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f125248b-5ef7-4283-bb8c-44549060fbb8-openshift-service-ca\") pod \"f125248b-5ef7-4283-bb8c-44549060fbb8\" (UID: \"f125248b-5ef7-4283-bb8c-44549060fbb8\") " Oct 30 01:47:47.089963 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:47.089888 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f125248b-5ef7-4283-bb8c-44549060fbb8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f125248b-5ef7-4283-bb8c-44549060fbb8-metrics-certs\") pod \"f125248b-5ef7-4283-bb8c-44549060fbb8\" (UID: \"f125248b-5ef7-4283-bb8c-44549060fbb8\") " Oct 30 01:47:47.090092 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:47.089995 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f125248b-5ef7-4283-bb8c-44549060fbb8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f125248b-5ef7-4283-bb8c-44549060fbb8-maas-api-tls\") pod \"f125248b-5ef7-4283-bb8c-44549060fbb8\" (UID: \"f125248b-5ef7-4283-bb8c-44549060fbb8\") " Oct 30 01:47:47.090664 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:47.090578 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f125248b-5ef7-4283-bb8c-44549060fbb8-openshift-service-ca" pod "f125248b-5ef7-4283-bb8c-44549060fbb8" (UID: "f125248b-5ef7-4283-bb8c-44549060fbb8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:47:47.093652 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:47.093583 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f125248b-5ef7-4283-bb8c-44549060fbb8-metrics-certs" pod "f125248b-5ef7-4283-bb8c-44549060fbb8" (UID: "f125248b-5ef7-4283-bb8c-44549060fbb8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:47:47.096091 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:47.096064 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f125248b-5ef7-4283-bb8c-44549060fbb8-maas-api-tls" pod "f125248b-5ef7-4283-bb8c-44549060fbb8" (UID: "f125248b-5ef7-4283-bb8c-44549060fbb8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:47:47.096374 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:47.096354 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f125248b-5ef7-4283-bb8c-44549060fbb8-kube-api-access-k4lxn" pod "f125248b-5ef7-4283-bb8c-44549060fbb8" (UID: "f125248b-5ef7-4283-bb8c-44549060fbb8"). InnerVolumeSpecName "kube-api-access-k4lxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:47:47.191665 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:47.191627 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4lxn\" (UniqueName: \"kubernetes.io/projected/f125248b-5ef7-4283-bb8c-44549060fbb8-kube-api-access-k4lxn\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:47:47.191665 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:47.191658 6221 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f125248b-5ef7-4283-bb8c-44549060fbb8-openshift-service-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:47:47.191665 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:47.191669 6221 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f125248b-5ef7-4283-bb8c-44549060fbb8-metrics-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:47:47.191665 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:47.191678 6221 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f125248b-5ef7-4283-bb8c-44549060fbb8-maas-api-tls\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:47:47.399562 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:47.396957 6221 generic.go:356] "Generic (PLEG): container finished" podID="f125248b-5ef7-4283-bb8c-44549060fbb8" containerID="cbf64338b43d6bf513539febcc6632d13138b9f1170bd1f062f8244166c28575" exitCode=0 Oct 30 01:47:47.399562 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:47.398452 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-58632b-58b9fff455-r2t2s" Oct 30 01:47:47.412505 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:47.411615 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-58632b-58b9fff455-r2t2s" event={"ID":"f125248b-5ef7-4283-bb8c-44549060fbb8","Type":"ContainerDied","Data":"cbf64338b43d6bf513539febcc6632d13138b9f1170bd1f062f8244166c28575"} Oct 30 01:47:47.412505 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:47.411714 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-58632b-58b9fff455-r2t2s" event={"ID":"f125248b-5ef7-4283-bb8c-44549060fbb8","Type":"ContainerDied","Data":"ea909b71d1c767d65b875f0ac10e50fb717cf9f29812681949d7489aecd6b4d4"} Oct 30 01:47:47.412505 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:47.411759 6221 scope.go:122] "RemoveContainer" containerID="cbf64338b43d6bf513539febcc6632d13138b9f1170bd1f062f8244166c28575" Oct 30 01:47:47.433737 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:47.433715 6221 scope.go:122] "RemoveContainer" containerID="cbf64338b43d6bf513539febcc6632d13138b9f1170bd1f062f8244166c28575" Oct 30 01:47:47.434133 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:47:47.434109 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbf64338b43d6bf513539febcc6632d13138b9f1170bd1f062f8244166c28575\": container with ID starting with cbf64338b43d6bf513539febcc6632d13138b9f1170bd1f062f8244166c28575 not found: ID does not exist" containerID="cbf64338b43d6bf513539febcc6632d13138b9f1170bd1f062f8244166c28575" Oct 30 01:47:47.434199 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:47.434146 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbf64338b43d6bf513539febcc6632d13138b9f1170bd1f062f8244166c28575"} err="failed to get container status \"cbf64338b43d6bf513539febcc6632d13138b9f1170bd1f062f8244166c28575\": rpc error: code = NotFound desc = could not find container \"cbf64338b43d6bf513539febcc6632d13138b9f1170bd1f062f8244166c28575\": container with ID starting with cbf64338b43d6bf513539febcc6632d13138b9f1170bd1f062f8244166c28575 not found: ID does not exist" Oct 30 01:47:47.467331 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:47.467291 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-58632b-58b9fff455-r2t2s"] Oct 30 01:47:47.479758 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:47.479719 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-58632b-58b9fff455-r2t2s"] Oct 30 01:47:48.548069 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:48.548034 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f125248b-5ef7-4283-bb8c-44549060fbb8" path="/var/lib/kubelet/pods/f125248b-5ef7-4283-bb8c-44549060fbb8/volumes" Oct 30 01:47:50.740135 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:50.740092 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6"] Oct 30 01:47:50.741180 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:50.741156 6221 cpu_manager.go:414] "containerMap: removing container" podUID="f125248b-5ef7-4283-bb8c-44549060fbb8" containerName="maas-api" Oct 30 01:47:50.741371 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:50.741332 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f125248b-5ef7-4283-bb8c-44549060fbb8" containerName="maas-api" Oct 30 01:47:50.741595 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:50.741575 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="f125248b-5ef7-4283-bb8c-44549060fbb8" containerName="maas-api" Oct 30 01:47:50.746723 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:50.746699 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" Oct 30 01:47:50.766417 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:50.766387 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-264bdd-openshift-default-dockercfg-8qt8s\"" Oct 30 01:47:50.771674 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:50.771593 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6"] Oct 30 01:47:50.919498 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:50.918909 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b685aa0-0c58-4911-8f31-3e920b8151a1-workload-socket\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6\" (UID: \"6b685aa0-0c58-4911-8f31-3e920b8151a1\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" Oct 30 01:47:50.919498 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:50.919022 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b685aa0-0c58-4911-8f31-3e920b8151a1-istio-data\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6\" (UID: \"6b685aa0-0c58-4911-8f31-3e920b8151a1\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" Oct 30 01:47:50.919498 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:50.919088 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b685aa0-0c58-4911-8f31-3e920b8151a1-istiod-ca-cert\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6\" (UID: \"6b685aa0-0c58-4911-8f31-3e920b8151a1\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" Oct 30 01:47:50.919498 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:50.919115 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b685aa0-0c58-4911-8f31-3e920b8151a1-istio-envoy\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6\" (UID: \"6b685aa0-0c58-4911-8f31-3e920b8151a1\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" Oct 30 01:47:50.919498 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:50.919219 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b685aa0-0c58-4911-8f31-3e920b8151a1-istio-podinfo\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6\" (UID: \"6b685aa0-0c58-4911-8f31-3e920b8151a1\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" Oct 30 01:47:50.919498 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:50.919268 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chljc\" (UniqueName: \"kubernetes.io/projected/6b685aa0-0c58-4911-8f31-3e920b8151a1-kube-api-access-chljc\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6\" (UID: \"6b685aa0-0c58-4911-8f31-3e920b8151a1\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" Oct 30 01:47:50.919498 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:50.919406 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b685aa0-0c58-4911-8f31-3e920b8151a1-credential-socket\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6\" (UID: \"6b685aa0-0c58-4911-8f31-3e920b8151a1\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" Oct 30 01:47:50.919498 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:50.919444 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b685aa0-0c58-4911-8f31-3e920b8151a1-istio-token\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6\" (UID: \"6b685aa0-0c58-4911-8f31-3e920b8151a1\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" Oct 30 01:47:50.920356 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:50.919609 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b685aa0-0c58-4911-8f31-3e920b8151a1-workload-certs\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6\" (UID: \"6b685aa0-0c58-4911-8f31-3e920b8151a1\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" Oct 30 01:47:51.039611 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:51.038562 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b685aa0-0c58-4911-8f31-3e920b8151a1-istio-data\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6\" (UID: \"6b685aa0-0c58-4911-8f31-3e920b8151a1\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" Oct 30 01:47:51.039611 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:51.038642 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b685aa0-0c58-4911-8f31-3e920b8151a1-istiod-ca-cert\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6\" (UID: \"6b685aa0-0c58-4911-8f31-3e920b8151a1\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" Oct 30 01:47:51.039611 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:51.038677 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b685aa0-0c58-4911-8f31-3e920b8151a1-istio-envoy\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6\" (UID: \"6b685aa0-0c58-4911-8f31-3e920b8151a1\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" Oct 30 01:47:51.039611 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:51.038776 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b685aa0-0c58-4911-8f31-3e920b8151a1-istio-podinfo\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6\" (UID: \"6b685aa0-0c58-4911-8f31-3e920b8151a1\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" Oct 30 01:47:51.039611 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:51.038814 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chljc\" (UniqueName: \"kubernetes.io/projected/6b685aa0-0c58-4911-8f31-3e920b8151a1-kube-api-access-chljc\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6\" (UID: \"6b685aa0-0c58-4911-8f31-3e920b8151a1\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" Oct 30 01:47:51.039611 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:51.038907 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b685aa0-0c58-4911-8f31-3e920b8151a1-credential-socket\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6\" (UID: \"6b685aa0-0c58-4911-8f31-3e920b8151a1\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" Oct 30 01:47:51.039611 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:51.038943 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b685aa0-0c58-4911-8f31-3e920b8151a1-istio-token\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6\" (UID: \"6b685aa0-0c58-4911-8f31-3e920b8151a1\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" Oct 30 01:47:51.039611 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:51.039042 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b685aa0-0c58-4911-8f31-3e920b8151a1-workload-certs\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6\" (UID: \"6b685aa0-0c58-4911-8f31-3e920b8151a1\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" Oct 30 01:47:51.039611 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:51.039090 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b685aa0-0c58-4911-8f31-3e920b8151a1-workload-socket\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6\" (UID: \"6b685aa0-0c58-4911-8f31-3e920b8151a1\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" Oct 30 01:47:51.040241 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:51.039675 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b685aa0-0c58-4911-8f31-3e920b8151a1-istiod-ca-cert\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6\" (UID: \"6b685aa0-0c58-4911-8f31-3e920b8151a1\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" Oct 30 01:47:51.066261 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:51.066220 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b685aa0-0c58-4911-8f31-3e920b8151a1-workload-socket\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6\" (UID: \"6b685aa0-0c58-4911-8f31-3e920b8151a1\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" Oct 30 01:47:51.069045 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:51.067228 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b685aa0-0c58-4911-8f31-3e920b8151a1-istio-data\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6\" (UID: \"6b685aa0-0c58-4911-8f31-3e920b8151a1\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" Oct 30 01:47:51.069261 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:51.068504 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b685aa0-0c58-4911-8f31-3e920b8151a1-workload-certs\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6\" (UID: \"6b685aa0-0c58-4911-8f31-3e920b8151a1\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" Oct 30 01:47:51.069504 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:51.068635 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b685aa0-0c58-4911-8f31-3e920b8151a1-credential-socket\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6\" (UID: \"6b685aa0-0c58-4911-8f31-3e920b8151a1\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" Oct 30 01:47:51.096349 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:51.096302 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b685aa0-0c58-4911-8f31-3e920b8151a1-istio-envoy\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6\" (UID: \"6b685aa0-0c58-4911-8f31-3e920b8151a1\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" Oct 30 01:47:51.110519 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:51.110478 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b685aa0-0c58-4911-8f31-3e920b8151a1-istio-podinfo\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6\" (UID: \"6b685aa0-0c58-4911-8f31-3e920b8151a1\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" Oct 30 01:47:51.112563 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:51.112472 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chljc\" (UniqueName: \"kubernetes.io/projected/6b685aa0-0c58-4911-8f31-3e920b8151a1-kube-api-access-chljc\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6\" (UID: \"6b685aa0-0c58-4911-8f31-3e920b8151a1\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" Oct 30 01:47:51.189488 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:51.189419 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b685aa0-0c58-4911-8f31-3e920b8151a1-istio-token\") pod \"e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6\" (UID: \"6b685aa0-0c58-4911-8f31-3e920b8151a1\") " pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" Oct 30 01:47:51.370941 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:51.370847 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" Oct 30 01:47:51.573897 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:47:51.573741 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:47:51.768492 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:51.768281 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6"] Oct 30 01:47:51.777959 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:51.777921 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:47:51.778071 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:51.777991 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:47:51.778071 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:51.778022 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:47:52.485461 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:52.485426 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" event={"ID":"6b685aa0-0c58-4911-8f31-3e920b8151a1","Type":"ContainerStarted","Data":"10902634aa9bb9a9f3db67ba6c2317fe28fee31b321eb17ddfaa4f4be3a78eb4"} Oct 30 01:47:52.485725 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:52.485701 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" event={"ID":"6b685aa0-0c58-4911-8f31-3e920b8151a1","Type":"ContainerStarted","Data":"814dd9f5f2e7b29fcf71d358efddcd4e918a5f5e062f894134794d8026fe12eb"} Oct 30 01:47:52.574384 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:52.573661 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" podStartSLOduration=2.573637825 podStartE2EDuration="2.573637825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:47:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:47:52.543498999 +0000 UTC m=+1796.121786335" watchObservedRunningTime="2026-10-30 01:47:52.573637825 +0000 UTC m=+1796.151925164" Oct 30 01:47:53.371641 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:53.371600 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" Oct 30 01:47:53.377262 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:53.377235 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" Oct 30 01:47:53.535555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:53.534822 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" Oct 30 01:47:53.545731 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:53.542867 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" Oct 30 01:47:55.393738 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:55.393707 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-861cd2-6477f778c6-8cbg5" Oct 30 01:47:56.803249 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:56.803221 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2vsns_d332bbb4-c699-44e8-96ba-516d8ca03e95/ovn-acl-logging/0.log" Oct 30 01:47:56.804820 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:56.804791 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2vsns_d332bbb4-c699-44e8-96ba-516d8ca03e95/ovn-acl-logging/0.log" Oct 30 01:47:56.922673 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:56.922625 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-264bdd-87b6c4ddc-ljngg"] Oct 30 01:47:56.927956 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:56.927931 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-264bdd-87b6c4ddc-ljngg" Oct 30 01:47:56.941357 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:56.941318 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-264bdd-87b6c4ddc-ljngg"] Oct 30 01:47:56.948866 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:56.948835 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-264bdd\"" Oct 30 01:47:56.957025 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:56.956992 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-264bdd-dockercfg-qbkjv\"" Oct 30 01:47:56.996346 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:56.996310 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9566007-9280-43dc-8c1d-0439ccc4f679-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-264bdd-87b6c4ddc-ljngg\" (UID: \"e9566007-9280-43dc-8c1d-0439ccc4f679\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-264bdd-87b6c4ddc-ljngg" Oct 30 01:47:56.996601 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:56.996573 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjj72\" (UniqueName: \"kubernetes.io/projected/e9566007-9280-43dc-8c1d-0439ccc4f679-kube-api-access-xjj72\") pod \"payload-pre-processing-e2e-shared-b-w2-264bdd-87b6c4ddc-ljngg\" (UID: \"e9566007-9280-43dc-8c1d-0439ccc4f679\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-264bdd-87b6c4ddc-ljngg" Oct 30 01:47:57.097921 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:57.097882 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjj72\" (UniqueName: \"kubernetes.io/projected/e9566007-9280-43dc-8c1d-0439ccc4f679-kube-api-access-xjj72\") pod \"payload-pre-processing-e2e-shared-b-w2-264bdd-87b6c4ddc-ljngg\" (UID: \"e9566007-9280-43dc-8c1d-0439ccc4f679\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-264bdd-87b6c4ddc-ljngg" Oct 30 01:47:57.098126 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:57.098064 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9566007-9280-43dc-8c1d-0439ccc4f679-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-264bdd-87b6c4ddc-ljngg\" (UID: \"e9566007-9280-43dc-8c1d-0439ccc4f679\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-264bdd-87b6c4ddc-ljngg" Oct 30 01:47:57.099016 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:57.098990 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9566007-9280-43dc-8c1d-0439ccc4f679-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-264bdd-87b6c4ddc-ljngg\" (UID: \"e9566007-9280-43dc-8c1d-0439ccc4f679\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-264bdd-87b6c4ddc-ljngg" Oct 30 01:47:57.120673 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:57.120396 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjj72\" (UniqueName: \"kubernetes.io/projected/e9566007-9280-43dc-8c1d-0439ccc4f679-kube-api-access-xjj72\") pod \"payload-pre-processing-e2e-shared-b-w2-264bdd-87b6c4ddc-ljngg\" (UID: \"e9566007-9280-43dc-8c1d-0439ccc4f679\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-264bdd-87b6c4ddc-ljngg" Oct 30 01:47:57.246697 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:57.246654 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-264bdd-87b6c4ddc-ljngg" Oct 30 01:47:57.669977 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:57.669947 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-264bdd-87b6c4ddc-ljngg"] Oct 30 01:47:57.674253 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:47:57.674214 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9566007_9280_43dc_8c1d_0439ccc4f679.slice/crio-4e83735212562bceaa84e18aefd00e017ccc8239b7144add9908df0e274dfc95 WatchSource:0}: Error finding container 4e83735212562bceaa84e18aefd00e017ccc8239b7144add9908df0e274dfc95: Status 404 returned error can't find the container with id 4e83735212562bceaa84e18aefd00e017ccc8239b7144add9908df0e274dfc95 Oct 30 01:47:58.644376 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:58.644333 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-264bdd-87b6c4ddc-ljngg" event={"ID":"e9566007-9280-43dc-8c1d-0439ccc4f679","Type":"ContainerStarted","Data":"ebb9f5fb42508a2ba90a6a966857fb802d567d5e94dc5c35e43fac2d4e563189"} Oct 30 01:47:58.644926 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:58.644390 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-264bdd-87b6c4ddc-ljngg" event={"ID":"e9566007-9280-43dc-8c1d-0439ccc4f679","Type":"ContainerStarted","Data":"4e83735212562bceaa84e18aefd00e017ccc8239b7144add9908df0e274dfc95"} Oct 30 01:47:58.644926 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:58.644427 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-264bdd-87b6c4ddc-ljngg" Oct 30 01:47:58.683065 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:47:58.683004 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-264bdd-87b6c4ddc-ljngg" podStartSLOduration=2.682986058 podStartE2EDuration="2.682986058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:47:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:47:58.68294931 +0000 UTC m=+1802.261236644" watchObservedRunningTime="2026-10-30 01:47:58.682986058 +0000 UTC m=+1802.261273396" Oct 30 01:48:00.663435 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:00.663398 6221 generic.go:356] "Generic (PLEG): container finished" podID="8e229797-d116-4670-86e9-e446396807ad" containerID="4a160237c9824cc92ad28b8c6d3c3961a6c0292b598bbf64354fc22f4219746b" exitCode=0 Oct 30 01:48:00.664139 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:00.663466 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" event={"ID":"8e229797-d116-4670-86e9-e446396807ad","Type":"ContainerDied","Data":"4a160237c9824cc92ad28b8c6d3c3961a6c0292b598bbf64354fc22f4219746b"} Oct 30 01:48:00.664139 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:00.663940 6221 scope.go:122] "RemoveContainer" containerID="4a160237c9824cc92ad28b8c6d3c3961a6c0292b598bbf64354fc22f4219746b" Oct 30 01:48:00.665800 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:00.665765 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:48:00.665921 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:00.665856 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:48:00.665921 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:00.665907 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:48:01.671302 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:01.671268 6221 generic.go:356] "Generic (PLEG): container finished" podID="fd152ecb-4d3c-494c-83c1-182f018a4db1" containerID="39aaa2dd6656f6d14ad55ea410fb92f75c423e6c041af570bed8c4a989d616b1" exitCode=0 Oct 30 01:48:01.671864 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:01.671336 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" event={"ID":"fd152ecb-4d3c-494c-83c1-182f018a4db1","Type":"ContainerDied","Data":"39aaa2dd6656f6d14ad55ea410fb92f75c423e6c041af570bed8c4a989d616b1"} Oct 30 01:48:01.671864 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:01.671839 6221 scope.go:122] "RemoveContainer" containerID="39aaa2dd6656f6d14ad55ea410fb92f75c423e6c041af570bed8c4a989d616b1" Oct 30 01:48:01.673699 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:01.673670 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:48:01.673816 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:01.673707 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" event={"ID":"8e229797-d116-4670-86e9-e446396807ad","Type":"ContainerStarted","Data":"61510b829862039589f3a3fcd126a95f6d4b12c782c30bc68dcb8748c3224fae"} Oct 30 01:48:01.673816 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:01.673741 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:48:01.673816 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:01.673776 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:48:02.287057 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:02.287022 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" Oct 30 01:48:02.288348 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:02.288323 6221 patch_prober.go:29] interesting pod/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 30 01:48:02.288480 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:02.288366 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" podUID="8e229797-d116-4670-86e9-e446396807ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 30 01:48:02.735614 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:02.732264 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-389dba-99dbf7cdd-8wpwx"] Oct 30 01:48:02.978612 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:02.978580 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-389dba-99dbf7cdd-8wpwx"] Oct 30 01:48:02.978612 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:02.978623 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" Oct 30 01:48:02.978875 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:02.978666 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" event={"ID":"fd152ecb-4d3c-494c-83c1-182f018a4db1","Type":"ContainerStarted","Data":"ea107ead421b1caa03cf70f9c55113a0a0667cb05b0215b77a8b54dc3edcf5d7"} Oct 30 01:48:02.978875 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:02.978729 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-389dba-99dbf7cdd-8wpwx" Oct 30 01:48:03.006369 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:03.006317 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-389dba\"" Oct 30 01:48:03.006369 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:03.006360 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-389dba-dockercfg-sghn6\"" Oct 30 01:48:03.096299 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:03.096253 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzdf6\" (UniqueName: \"kubernetes.io/projected/cbf9b943-008e-4887-abef-556d30970ff5-kube-api-access-wzdf6\") pod \"payload-processing-e2e-explicit-389dba-99dbf7cdd-8wpwx\" (UID: \"cbf9b943-008e-4887-abef-556d30970ff5\") " pod="openshift-ingress/payload-processing-e2e-explicit-389dba-99dbf7cdd-8wpwx" Oct 30 01:48:03.096712 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:03.096681 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbf9b943-008e-4887-abef-556d30970ff5-plugins-config-volume\") pod \"payload-processing-e2e-explicit-389dba-99dbf7cdd-8wpwx\" (UID: \"cbf9b943-008e-4887-abef-556d30970ff5\") " pod="openshift-ingress/payload-processing-e2e-explicit-389dba-99dbf7cdd-8wpwx" Oct 30 01:48:03.198503 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:03.198453 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzdf6\" (UniqueName: \"kubernetes.io/projected/cbf9b943-008e-4887-abef-556d30970ff5-kube-api-access-wzdf6\") pod \"payload-processing-e2e-explicit-389dba-99dbf7cdd-8wpwx\" (UID: \"cbf9b943-008e-4887-abef-556d30970ff5\") " pod="openshift-ingress/payload-processing-e2e-explicit-389dba-99dbf7cdd-8wpwx" Oct 30 01:48:03.199253 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:03.199226 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbf9b943-008e-4887-abef-556d30970ff5-plugins-config-volume\") pod \"payload-processing-e2e-explicit-389dba-99dbf7cdd-8wpwx\" (UID: \"cbf9b943-008e-4887-abef-556d30970ff5\") " pod="openshift-ingress/payload-processing-e2e-explicit-389dba-99dbf7cdd-8wpwx" Oct 30 01:48:03.211559 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:03.200289 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbf9b943-008e-4887-abef-556d30970ff5-plugins-config-volume\") pod \"payload-processing-e2e-explicit-389dba-99dbf7cdd-8wpwx\" (UID: \"cbf9b943-008e-4887-abef-556d30970ff5\") " pod="openshift-ingress/payload-processing-e2e-explicit-389dba-99dbf7cdd-8wpwx" Oct 30 01:48:03.229622 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:03.221577 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzdf6\" (UniqueName: \"kubernetes.io/projected/cbf9b943-008e-4887-abef-556d30970ff5-kube-api-access-wzdf6\") pod \"payload-processing-e2e-explicit-389dba-99dbf7cdd-8wpwx\" (UID: \"cbf9b943-008e-4887-abef-556d30970ff5\") " pod="openshift-ingress/payload-processing-e2e-explicit-389dba-99dbf7cdd-8wpwx" Oct 30 01:48:03.320556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:03.305791 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-389dba-99dbf7cdd-8wpwx" Oct 30 01:48:03.386558 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:03.383866 6221 patch_prober.go:29] interesting pod/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 30 01:48:03.386558 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:03.383939 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" podUID="8e229797-d116-4670-86e9-e446396807ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 30 01:48:03.680623 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:03.680276 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" Oct 30 01:48:03.692235 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:03.691558 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" Oct 30 01:48:03.695956 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:03.695768 6221 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 01:48:03.695956 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:03.695844 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" podUID="fd152ecb-4d3c-494c-83c1-182f018a4db1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 01:48:03.963363 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:03.963321 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-389dba-99dbf7cdd-8wpwx"] Oct 30 01:48:03.965824 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:48:03.964924 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbf9b943_008e_4887_abef_556d30970ff5.slice/crio-87c4892ccfce186cc5e593c8e8778c3a023f9966b2e46d6b67aa8864865a0f22 WatchSource:0}: Error finding container 87c4892ccfce186cc5e593c8e8778c3a023f9966b2e46d6b67aa8864865a0f22: Status 404 returned error can't find the container with id 87c4892ccfce186cc5e593c8e8778c3a023f9966b2e46d6b67aa8864865a0f22 Oct 30 01:48:04.286486 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:04.286455 6221 patch_prober.go:29] interesting pod/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 30 01:48:04.286666 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:04.286507 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" podUID="8e229797-d116-4670-86e9-e446396807ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 30 01:48:04.689784 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:04.689686 6221 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 01:48:04.689784 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:04.689738 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" podUID="fd152ecb-4d3c-494c-83c1-182f018a4db1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 01:48:04.903013 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:04.902977 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-389dba-99dbf7cdd-8wpwx" event={"ID":"cbf9b943-008e-4887-abef-556d30970ff5","Type":"ContainerStarted","Data":"d006451191c5b404351b3408076f0e593b4f17312d946cf7aa9d602791e1aafa"} Oct 30 01:48:04.903199 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:04.903036 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-389dba-99dbf7cdd-8wpwx" Oct 30 01:48:04.903199 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:04.903058 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-389dba-99dbf7cdd-8wpwx" event={"ID":"cbf9b943-008e-4887-abef-556d30970ff5","Type":"ContainerStarted","Data":"87c4892ccfce186cc5e593c8e8778c3a023f9966b2e46d6b67aa8864865a0f22"} Oct 30 01:48:04.948618 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:04.948448 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-389dba-99dbf7cdd-8wpwx" podStartSLOduration=2.948425874 podStartE2EDuration="2.948425874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:48:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:48:04.944985731 +0000 UTC m=+1808.523273068" watchObservedRunningTime="2026-10-30 01:48:04.948425874 +0000 UTC m=+1808.526713211" Oct 30 01:48:05.287036 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:05.286995 6221 patch_prober.go:29] interesting pod/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 30 01:48:05.287587 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:05.287559 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" podUID="8e229797-d116-4670-86e9-e446396807ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 30 01:48:05.689232 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:05.689133 6221 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 01:48:05.689232 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:05.689204 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" podUID="fd152ecb-4d3c-494c-83c1-182f018a4db1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 01:48:06.277323 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:06.277280 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d"] Oct 30 01:48:06.282736 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:06.281664 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d"] Oct 30 01:48:06.282736 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:06.281757 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" Oct 30 01:48:06.287656 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:06.287628 6221 patch_prober.go:29] interesting pod/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 30 01:48:06.288014 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:06.287678 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" podUID="8e229797-d116-4670-86e9-e446396807ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 30 01:48:06.297722 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:06.297691 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-657fba-openshift-default-dockercfg-vwt4n\"" Oct 30 01:48:06.356262 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:06.356217 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56476948-dc26-41d2-9df4-e23f8a293d75-istio-token\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d\" (UID: \"56476948-dc26-41d2-9df4-e23f8a293d75\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" Oct 30 01:48:06.356595 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:06.356567 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56476948-dc26-41d2-9df4-e23f8a293d75-istio-podinfo\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d\" (UID: \"56476948-dc26-41d2-9df4-e23f8a293d75\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" Oct 30 01:48:06.356772 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:06.356754 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56476948-dc26-41d2-9df4-e23f8a293d75-istio-data\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d\" (UID: \"56476948-dc26-41d2-9df4-e23f8a293d75\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" Oct 30 01:48:06.356903 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:06.356887 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwc7f\" (UniqueName: \"kubernetes.io/projected/56476948-dc26-41d2-9df4-e23f8a293d75-kube-api-access-rwc7f\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d\" (UID: \"56476948-dc26-41d2-9df4-e23f8a293d75\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" Oct 30 01:48:06.357080 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:06.357064 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56476948-dc26-41d2-9df4-e23f8a293d75-workload-socket\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d\" (UID: \"56476948-dc26-41d2-9df4-e23f8a293d75\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" Oct 30 01:48:06.360654 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:06.360624 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56476948-dc26-41d2-9df4-e23f8a293d75-credential-socket\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d\" (UID: \"56476948-dc26-41d2-9df4-e23f8a293d75\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" Oct 30 01:48:06.360840 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:06.360821 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56476948-dc26-41d2-9df4-e23f8a293d75-workload-certs\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d\" (UID: \"56476948-dc26-41d2-9df4-e23f8a293d75\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" Oct 30 01:48:06.360987 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:06.360968 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56476948-dc26-41d2-9df4-e23f8a293d75-istio-envoy\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d\" (UID: \"56476948-dc26-41d2-9df4-e23f8a293d75\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" Oct 30 01:48:06.361117 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:06.361101 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56476948-dc26-41d2-9df4-e23f8a293d75-istiod-ca-cert\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d\" (UID: \"56476948-dc26-41d2-9df4-e23f8a293d75\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" Oct 30 01:48:06.481661 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:06.472322 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56476948-dc26-41d2-9df4-e23f8a293d75-istio-podinfo\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d\" (UID: \"56476948-dc26-41d2-9df4-e23f8a293d75\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" Oct 30 01:48:06.522551 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:06.502880 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56476948-dc26-41d2-9df4-e23f8a293d75-istio-data\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d\" (UID: \"56476948-dc26-41d2-9df4-e23f8a293d75\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" Oct 30 01:48:06.522551 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:06.502951 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwc7f\" (UniqueName: \"kubernetes.io/projected/56476948-dc26-41d2-9df4-e23f8a293d75-kube-api-access-rwc7f\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d\" (UID: \"56476948-dc26-41d2-9df4-e23f8a293d75\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" Oct 30 01:48:06.522551 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:06.503088 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56476948-dc26-41d2-9df4-e23f8a293d75-workload-socket\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d\" (UID: \"56476948-dc26-41d2-9df4-e23f8a293d75\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" Oct 30 01:48:06.522551 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:06.503171 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56476948-dc26-41d2-9df4-e23f8a293d75-credential-socket\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d\" (UID: \"56476948-dc26-41d2-9df4-e23f8a293d75\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" Oct 30 01:48:06.522551 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:06.503203 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56476948-dc26-41d2-9df4-e23f8a293d75-workload-certs\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d\" (UID: \"56476948-dc26-41d2-9df4-e23f8a293d75\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" Oct 30 01:48:06.522551 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:06.503248 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56476948-dc26-41d2-9df4-e23f8a293d75-istio-envoy\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d\" (UID: \"56476948-dc26-41d2-9df4-e23f8a293d75\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" Oct 30 01:48:06.522551 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:06.503278 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56476948-dc26-41d2-9df4-e23f8a293d75-istiod-ca-cert\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d\" (UID: \"56476948-dc26-41d2-9df4-e23f8a293d75\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" Oct 30 01:48:06.522551 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:06.503605 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56476948-dc26-41d2-9df4-e23f8a293d75-istio-token\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d\" (UID: \"56476948-dc26-41d2-9df4-e23f8a293d75\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" Oct 30 01:48:06.522551 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:06.504428 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56476948-dc26-41d2-9df4-e23f8a293d75-istio-data\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d\" (UID: \"56476948-dc26-41d2-9df4-e23f8a293d75\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" Oct 30 01:48:06.522551 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:06.504986 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56476948-dc26-41d2-9df4-e23f8a293d75-workload-socket\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d\" (UID: \"56476948-dc26-41d2-9df4-e23f8a293d75\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" Oct 30 01:48:06.522551 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:06.505259 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56476948-dc26-41d2-9df4-e23f8a293d75-credential-socket\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d\" (UID: \"56476948-dc26-41d2-9df4-e23f8a293d75\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" Oct 30 01:48:06.522551 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:06.505549 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56476948-dc26-41d2-9df4-e23f8a293d75-workload-certs\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d\" (UID: \"56476948-dc26-41d2-9df4-e23f8a293d75\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" Oct 30 01:48:06.522551 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:06.512249 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56476948-dc26-41d2-9df4-e23f8a293d75-istiod-ca-cert\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d\" (UID: \"56476948-dc26-41d2-9df4-e23f8a293d75\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" Oct 30 01:48:06.610643 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:06.610608 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwc7f\" (UniqueName: \"kubernetes.io/projected/56476948-dc26-41d2-9df4-e23f8a293d75-kube-api-access-rwc7f\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d\" (UID: \"56476948-dc26-41d2-9df4-e23f8a293d75\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" Oct 30 01:48:06.626261 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:06.626223 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56476948-dc26-41d2-9df4-e23f8a293d75-istio-podinfo\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d\" (UID: \"56476948-dc26-41d2-9df4-e23f8a293d75\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" Oct 30 01:48:06.626930 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:06.626906 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56476948-dc26-41d2-9df4-e23f8a293d75-istio-envoy\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d\" (UID: \"56476948-dc26-41d2-9df4-e23f8a293d75\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" Oct 30 01:48:06.633082 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:06.633051 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56476948-dc26-41d2-9df4-e23f8a293d75-istio-token\") pod \"e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d\" (UID: \"56476948-dc26-41d2-9df4-e23f8a293d75\") " pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" Oct 30 01:48:06.658094 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:06.658060 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" Oct 30 01:48:06.709505 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:06.709467 6221 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 01:48:06.709861 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:06.709809 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" podUID="fd152ecb-4d3c-494c-83c1-182f018a4db1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 01:48:07.155031 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:07.154990 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d"] Oct 30 01:48:07.162141 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:48:07.161635 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56476948_dc26_41d2_9df4_e23f8a293d75.slice/crio-cfff332c899cf7186306218413a0d2e099ba7e61aaea6dc6a47b5b8fe8fe3f8a WatchSource:0}: Error finding container cfff332c899cf7186306218413a0d2e099ba7e61aaea6dc6a47b5b8fe8fe3f8a: Status 404 returned error can't find the container with id cfff332c899cf7186306218413a0d2e099ba7e61aaea6dc6a47b5b8fe8fe3f8a Oct 30 01:48:07.203664 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:07.184816 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:48:07.203664 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:07.184921 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:48:07.203664 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:07.184963 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:48:07.286906 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:07.286878 6221 patch_prober.go:29] interesting pod/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body= Oct 30 01:48:07.287053 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:07.287030 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" podUID="8e229797-d116-4670-86e9-e446396807ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" Oct 30 01:48:07.689457 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:07.689103 6221 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 30 01:48:07.689457 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:07.689170 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" podUID="fd152ecb-4d3c-494c-83c1-182f018a4db1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 30 01:48:07.953054 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:07.952915 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" event={"ID":"56476948-dc26-41d2-9df4-e23f8a293d75","Type":"ContainerStarted","Data":"e19c9fa6d3979b03b5ef652ad07285786196e7158988e3dd4c87217b0087e747"} Oct 30 01:48:07.953054 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:07.952974 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" event={"ID":"56476948-dc26-41d2-9df4-e23f8a293d75","Type":"ContainerStarted","Data":"cfff332c899cf7186306218413a0d2e099ba7e61aaea6dc6a47b5b8fe8fe3f8a"} Oct 30 01:48:08.002513 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:08.002449 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" podStartSLOduration=2.002427945 podStartE2EDuration="2.002427945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:48:07.999781749 +0000 UTC m=+1811.578069086" watchObservedRunningTime="2026-10-30 01:48:08.002427945 +0000 UTC m=+1811.580715282" Oct 30 01:48:08.291238 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:08.291207 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" Oct 30 01:48:08.292389 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:08.292367 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" Oct 30 01:48:08.659197 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:08.659099 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" Oct 30 01:48:08.666456 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:08.666426 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" Oct 30 01:48:08.702257 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:08.702218 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" Oct 30 01:48:08.703903 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:08.703876 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" Oct 30 01:48:08.959222 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:08.959129 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" Oct 30 01:48:08.960156 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:08.960127 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" Oct 30 01:48:09.656393 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:09.656357 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-264bdd-87b6c4ddc-ljngg" Oct 30 01:48:14.306665 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:14.306627 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:48:15.938020 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:15.937183 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-389dba-99dbf7cdd-8wpwx" Oct 30 01:48:21.500302 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:48:21.500266 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:48:22.655807 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:22.655762 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-c8d376/auth-test-model-c8d376-kserve-6bddf577bf-jw4k8"] Oct 30 01:48:22.662171 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:22.662144 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-c8d376/auth-test-model-c8d376-kserve-6bddf577bf-jw4k8" Oct 30 01:48:22.679980 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:22.679949 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c8d376/auth-test-model-c8d376-kserve-6bddf577bf-jw4k8"] Oct 30 01:48:22.685809 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:22.685776 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-c8d376\"/\"default-dockercfg-q7m5j\"" Oct 30 01:48:22.685975 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:22.685819 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-c8d376\"/\"openshift-service-ca.crt\"" Oct 30 01:48:22.698749 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:22.698705 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-c8d376\"/\"auth-test-model-c8d376-kserve-self-signed-certs\"" Oct 30 01:48:22.698920 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:22.698828 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-c8d376\"/\"kube-root-ca.crt\"" Oct 30 01:48:22.774548 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:22.774507 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l6b5\" (UniqueName: \"kubernetes.io/projected/2a8dd470-155e-4470-b5e5-01ee5c9a0065-kube-api-access-7l6b5\") pod \"auth-test-model-c8d376-kserve-6bddf577bf-jw4k8\" (UID: \"2a8dd470-155e-4470-b5e5-01ee5c9a0065\") " pod="ai-tenant-e2e-shared-b-w0-c8d376/auth-test-model-c8d376-kserve-6bddf577bf-jw4k8" Oct 30 01:48:22.774741 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:22.774632 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a8dd470-155e-4470-b5e5-01ee5c9a0065-tls-certs\") pod \"auth-test-model-c8d376-kserve-6bddf577bf-jw4k8\" (UID: \"2a8dd470-155e-4470-b5e5-01ee5c9a0065\") " pod="ai-tenant-e2e-shared-b-w0-c8d376/auth-test-model-c8d376-kserve-6bddf577bf-jw4k8" Oct 30 01:48:22.774741 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:22.774693 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a8dd470-155e-4470-b5e5-01ee5c9a0065-dshm\") pod \"auth-test-model-c8d376-kserve-6bddf577bf-jw4k8\" (UID: \"2a8dd470-155e-4470-b5e5-01ee5c9a0065\") " pod="ai-tenant-e2e-shared-b-w0-c8d376/auth-test-model-c8d376-kserve-6bddf577bf-jw4k8" Oct 30 01:48:22.774837 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:22.774769 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a8dd470-155e-4470-b5e5-01ee5c9a0065-tmp-dir\") pod \"auth-test-model-c8d376-kserve-6bddf577bf-jw4k8\" (UID: \"2a8dd470-155e-4470-b5e5-01ee5c9a0065\") " pod="ai-tenant-e2e-shared-b-w0-c8d376/auth-test-model-c8d376-kserve-6bddf577bf-jw4k8" Oct 30 01:48:22.774837 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:22.774802 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a8dd470-155e-4470-b5e5-01ee5c9a0065-model-cache\") pod \"auth-test-model-c8d376-kserve-6bddf577bf-jw4k8\" (UID: \"2a8dd470-155e-4470-b5e5-01ee5c9a0065\") " pod="ai-tenant-e2e-shared-b-w0-c8d376/auth-test-model-c8d376-kserve-6bddf577bf-jw4k8" Oct 30 01:48:22.774913 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:22.774883 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a8dd470-155e-4470-b5e5-01ee5c9a0065-home\") pod \"auth-test-model-c8d376-kserve-6bddf577bf-jw4k8\" (UID: \"2a8dd470-155e-4470-b5e5-01ee5c9a0065\") " pod="ai-tenant-e2e-shared-b-w0-c8d376/auth-test-model-c8d376-kserve-6bddf577bf-jw4k8" Oct 30 01:48:22.876294 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:22.876246 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l6b5\" (UniqueName: \"kubernetes.io/projected/2a8dd470-155e-4470-b5e5-01ee5c9a0065-kube-api-access-7l6b5\") pod \"auth-test-model-c8d376-kserve-6bddf577bf-jw4k8\" (UID: \"2a8dd470-155e-4470-b5e5-01ee5c9a0065\") " pod="ai-tenant-e2e-shared-b-w0-c8d376/auth-test-model-c8d376-kserve-6bddf577bf-jw4k8" Oct 30 01:48:22.876479 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:22.876339 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a8dd470-155e-4470-b5e5-01ee5c9a0065-tls-certs\") pod \"auth-test-model-c8d376-kserve-6bddf577bf-jw4k8\" (UID: \"2a8dd470-155e-4470-b5e5-01ee5c9a0065\") " pod="ai-tenant-e2e-shared-b-w0-c8d376/auth-test-model-c8d376-kserve-6bddf577bf-jw4k8" Oct 30 01:48:22.876479 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:22.876363 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a8dd470-155e-4470-b5e5-01ee5c9a0065-dshm\") pod \"auth-test-model-c8d376-kserve-6bddf577bf-jw4k8\" (UID: \"2a8dd470-155e-4470-b5e5-01ee5c9a0065\") " pod="ai-tenant-e2e-shared-b-w0-c8d376/auth-test-model-c8d376-kserve-6bddf577bf-jw4k8" Oct 30 01:48:22.876479 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:22.876432 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a8dd470-155e-4470-b5e5-01ee5c9a0065-tmp-dir\") pod \"auth-test-model-c8d376-kserve-6bddf577bf-jw4k8\" (UID: \"2a8dd470-155e-4470-b5e5-01ee5c9a0065\") " pod="ai-tenant-e2e-shared-b-w0-c8d376/auth-test-model-c8d376-kserve-6bddf577bf-jw4k8" Oct 30 01:48:22.876479 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:22.876454 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a8dd470-155e-4470-b5e5-01ee5c9a0065-model-cache\") pod \"auth-test-model-c8d376-kserve-6bddf577bf-jw4k8\" (UID: \"2a8dd470-155e-4470-b5e5-01ee5c9a0065\") " pod="ai-tenant-e2e-shared-b-w0-c8d376/auth-test-model-c8d376-kserve-6bddf577bf-jw4k8" Oct 30 01:48:22.876744 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:22.876482 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a8dd470-155e-4470-b5e5-01ee5c9a0065-home\") pod \"auth-test-model-c8d376-kserve-6bddf577bf-jw4k8\" (UID: \"2a8dd470-155e-4470-b5e5-01ee5c9a0065\") " pod="ai-tenant-e2e-shared-b-w0-c8d376/auth-test-model-c8d376-kserve-6bddf577bf-jw4k8" Oct 30 01:48:22.876971 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:22.876947 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a8dd470-155e-4470-b5e5-01ee5c9a0065-home\") pod \"auth-test-model-c8d376-kserve-6bddf577bf-jw4k8\" (UID: \"2a8dd470-155e-4470-b5e5-01ee5c9a0065\") " pod="ai-tenant-e2e-shared-b-w0-c8d376/auth-test-model-c8d376-kserve-6bddf577bf-jw4k8" Oct 30 01:48:22.877060 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:22.877023 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a8dd470-155e-4470-b5e5-01ee5c9a0065-tmp-dir\") pod \"auth-test-model-c8d376-kserve-6bddf577bf-jw4k8\" (UID: \"2a8dd470-155e-4470-b5e5-01ee5c9a0065\") " pod="ai-tenant-e2e-shared-b-w0-c8d376/auth-test-model-c8d376-kserve-6bddf577bf-jw4k8" Oct 30 01:48:22.877121 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:22.877092 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a8dd470-155e-4470-b5e5-01ee5c9a0065-model-cache\") pod \"auth-test-model-c8d376-kserve-6bddf577bf-jw4k8\" (UID: \"2a8dd470-155e-4470-b5e5-01ee5c9a0065\") " pod="ai-tenant-e2e-shared-b-w0-c8d376/auth-test-model-c8d376-kserve-6bddf577bf-jw4k8" Oct 30 01:48:22.878849 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:22.878824 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a8dd470-155e-4470-b5e5-01ee5c9a0065-dshm\") pod \"auth-test-model-c8d376-kserve-6bddf577bf-jw4k8\" (UID: \"2a8dd470-155e-4470-b5e5-01ee5c9a0065\") " pod="ai-tenant-e2e-shared-b-w0-c8d376/auth-test-model-c8d376-kserve-6bddf577bf-jw4k8" Oct 30 01:48:22.879256 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:22.879232 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a8dd470-155e-4470-b5e5-01ee5c9a0065-tls-certs\") pod \"auth-test-model-c8d376-kserve-6bddf577bf-jw4k8\" (UID: \"2a8dd470-155e-4470-b5e5-01ee5c9a0065\") " pod="ai-tenant-e2e-shared-b-w0-c8d376/auth-test-model-c8d376-kserve-6bddf577bf-jw4k8" Oct 30 01:48:22.900134 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:22.900106 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l6b5\" (UniqueName: \"kubernetes.io/projected/2a8dd470-155e-4470-b5e5-01ee5c9a0065-kube-api-access-7l6b5\") pod \"auth-test-model-c8d376-kserve-6bddf577bf-jw4k8\" (UID: \"2a8dd470-155e-4470-b5e5-01ee5c9a0065\") " pod="ai-tenant-e2e-shared-b-w0-c8d376/auth-test-model-c8d376-kserve-6bddf577bf-jw4k8" Oct 30 01:48:22.985547 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:22.985450 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-c8d376/auth-test-model-c8d376-kserve-6bddf577bf-jw4k8" Oct 30 01:48:23.427702 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:23.427674 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c8d376/auth-test-model-c8d376-kserve-6bddf577bf-jw4k8"] Oct 30 01:48:23.430407 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:48:23.430374 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a8dd470_155e_4470_b5e5_01ee5c9a0065.slice/crio-0fb2407e7530decd2032dfaee6bdcadc2e8b7dcea0742728766c5017410dd362 WatchSource:0}: Error finding container 0fb2407e7530decd2032dfaee6bdcadc2e8b7dcea0742728766c5017410dd362: Status 404 returned error can't find the container with id 0fb2407e7530decd2032dfaee6bdcadc2e8b7dcea0742728766c5017410dd362 Oct 30 01:48:24.138740 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:24.138687 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-c8d376/auth-test-model-c8d376-kserve-6bddf577bf-jw4k8" event={"ID":"2a8dd470-155e-4470-b5e5-01ee5c9a0065","Type":"ContainerStarted","Data":"313066bfe22113b498f4b83cebbe2e9f05f5ab1ca17c4b5297206a6f65ec8344"} Oct 30 01:48:24.139173 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:24.138749 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-c8d376/auth-test-model-c8d376-kserve-6bddf577bf-jw4k8" event={"ID":"2a8dd470-155e-4470-b5e5-01ee5c9a0065","Type":"ContainerStarted","Data":"0fb2407e7530decd2032dfaee6bdcadc2e8b7dcea0742728766c5017410dd362"} Oct 30 01:48:24.178923 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:24.178843 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-c8d376/auth-test-model-c8d376-kserve-6bddf577bf-jw4k8" podStartSLOduration=2.1788223589999998 podStartE2EDuration="2.178822359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:48:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:48:24.177200097 +0000 UTC m=+1827.755487434" watchObservedRunningTime="2026-10-30 01:48:24.178822359 +0000 UTC m=+1827.757109697" Oct 30 01:48:24.986350 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:24.986316 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-c8d376/auth-test-model-c8d376-kserve-6bddf577bf-jw4k8" Oct 30 01:48:25.225391 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:25.224127 6221 patch_prober.go:29] interesting pod/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:48:25.225391 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:25.224184 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" podUID="51dc5d9e-31aa-4246-9bd6-8953c55424f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:48:26.153229 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:26.153194 6221 generic.go:356] "Generic (PLEG): container finished" podID="51dc5d9e-31aa-4246-9bd6-8953c55424f1" containerID="43303036d7d0a45bfc51c6bd52645cd1b9c19a912ac61ad1fe416e6f4e6d3baa" exitCode=0 Oct 30 01:48:26.153440 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:26.153270 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" event={"ID":"51dc5d9e-31aa-4246-9bd6-8953c55424f1","Type":"ContainerDied","Data":"43303036d7d0a45bfc51c6bd52645cd1b9c19a912ac61ad1fe416e6f4e6d3baa"} Oct 30 01:48:26.153983 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:26.153965 6221 scope.go:122] "RemoveContainer" containerID="43303036d7d0a45bfc51c6bd52645cd1b9c19a912ac61ad1fe416e6f4e6d3baa" Oct 30 01:48:26.155655 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:26.155626 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:48:26.155772 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:26.155711 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:48:26.155772 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:26.155762 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:48:27.162371 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:27.162334 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" event={"ID":"51dc5d9e-31aa-4246-9bd6-8953c55424f1","Type":"ContainerStarted","Data":"453e8271f6f7cc307c88daa03eff05c7b1047b9e97e07910e9b740c235b58d7e"} Oct 30 01:48:27.231556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:27.230943 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" Oct 30 01:48:27.231556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:27.231076 6221 patch_prober.go:29] interesting pod/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 01:48:27.231556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:27.231119 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" podUID="51dc5d9e-31aa-4246-9bd6-8953c55424f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 01:48:28.228555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:28.227933 6221 patch_prober.go:29] interesting pod/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 01:48:28.228555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:28.227998 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" podUID="51dc5d9e-31aa-4246-9bd6-8953c55424f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 01:48:29.228555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:29.227945 6221 patch_prober.go:29] interesting pod/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 01:48:29.228555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:29.228009 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" podUID="51dc5d9e-31aa-4246-9bd6-8953c55424f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 01:48:30.222971 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:30.222930 6221 patch_prober.go:29] interesting pod/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 01:48:30.223284 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:30.223252 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" podUID="51dc5d9e-31aa-4246-9bd6-8953c55424f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 01:48:31.222556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:31.220975 6221 patch_prober.go:29] interesting pod/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 01:48:31.222556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:31.221056 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" podUID="51dc5d9e-31aa-4246-9bd6-8953c55424f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 01:48:32.222844 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:32.222801 6221 patch_prober.go:29] interesting pod/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body= Oct 30 01:48:32.223492 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:32.223454 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" podUID="51dc5d9e-31aa-4246-9bd6-8953c55424f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" Oct 30 01:48:32.986382 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:32.986283 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-c8d376/auth-test-model-c8d376-kserve-6bddf577bf-jw4k8" Oct 30 01:48:32.999495 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:32.999466 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-c8d376/auth-test-model-c8d376-kserve-6bddf577bf-jw4k8" Oct 30 01:48:33.214947 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:33.214919 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-c8d376/auth-test-model-c8d376-kserve-6bddf577bf-jw4k8" Oct 30 01:48:33.225743 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:33.225714 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" Oct 30 01:48:33.226239 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:33.226007 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" Oct 30 01:48:33.226707 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:33.226689 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" Oct 30 01:48:35.006281 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:35.006248 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-657fba-666fc89fb4-tc5q4"] Oct 30 01:48:35.010696 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:35.010677 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-657fba-666fc89fb4-tc5q4" Oct 30 01:48:35.025195 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:35.025174 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-657fba-serving-cert\"" Oct 30 01:48:35.040997 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:35.040960 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-657fba-666fc89fb4-tc5q4"] Oct 30 01:48:35.175991 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:35.175948 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5b94b56-1785-45b7-ae04-3afb97ace8a3-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-657fba-666fc89fb4-tc5q4\" (UID: \"d5b94b56-1785-45b7-ae04-3afb97ace8a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-657fba-666fc89fb4-tc5q4" Oct 30 01:48:35.176198 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:35.176053 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2b8r\" (UniqueName: \"kubernetes.io/projected/d5b94b56-1785-45b7-ae04-3afb97ace8a3-kube-api-access-l2b8r\") pod \"maas-api-e2e-shared-b-w4-657fba-666fc89fb4-tc5q4\" (UID: \"d5b94b56-1785-45b7-ae04-3afb97ace8a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-657fba-666fc89fb4-tc5q4" Oct 30 01:48:35.176198 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:35.176121 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5b94b56-1785-45b7-ae04-3afb97ace8a3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-657fba-666fc89fb4-tc5q4\" (UID: \"d5b94b56-1785-45b7-ae04-3afb97ace8a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-657fba-666fc89fb4-tc5q4" Oct 30 01:48:35.176329 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:35.176290 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5b94b56-1785-45b7-ae04-3afb97ace8a3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-657fba-666fc89fb4-tc5q4\" (UID: \"d5b94b56-1785-45b7-ae04-3afb97ace8a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-657fba-666fc89fb4-tc5q4" Oct 30 01:48:35.284446 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:35.283740 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5b94b56-1785-45b7-ae04-3afb97ace8a3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-657fba-666fc89fb4-tc5q4\" (UID: \"d5b94b56-1785-45b7-ae04-3afb97ace8a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-657fba-666fc89fb4-tc5q4" Oct 30 01:48:35.284446 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:35.283990 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5b94b56-1785-45b7-ae04-3afb97ace8a3-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-657fba-666fc89fb4-tc5q4\" (UID: \"d5b94b56-1785-45b7-ae04-3afb97ace8a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-657fba-666fc89fb4-tc5q4" Oct 30 01:48:35.284446 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:35.284016 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2b8r\" (UniqueName: \"kubernetes.io/projected/d5b94b56-1785-45b7-ae04-3afb97ace8a3-kube-api-access-l2b8r\") pod \"maas-api-e2e-shared-b-w4-657fba-666fc89fb4-tc5q4\" (UID: \"d5b94b56-1785-45b7-ae04-3afb97ace8a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-657fba-666fc89fb4-tc5q4" Oct 30 01:48:35.284446 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:35.284070 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5b94b56-1785-45b7-ae04-3afb97ace8a3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-657fba-666fc89fb4-tc5q4\" (UID: \"d5b94b56-1785-45b7-ae04-3afb97ace8a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-657fba-666fc89fb4-tc5q4" Oct 30 01:48:35.286043 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:35.286006 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5b94b56-1785-45b7-ae04-3afb97ace8a3-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-657fba-666fc89fb4-tc5q4\" (UID: \"d5b94b56-1785-45b7-ae04-3afb97ace8a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-657fba-666fc89fb4-tc5q4" Oct 30 01:48:35.291563 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:35.289102 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5b94b56-1785-45b7-ae04-3afb97ace8a3-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-657fba-666fc89fb4-tc5q4\" (UID: \"d5b94b56-1785-45b7-ae04-3afb97ace8a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-657fba-666fc89fb4-tc5q4" Oct 30 01:48:35.291563 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:35.289135 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5b94b56-1785-45b7-ae04-3afb97ace8a3-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-657fba-666fc89fb4-tc5q4\" (UID: \"d5b94b56-1785-45b7-ae04-3afb97ace8a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-657fba-666fc89fb4-tc5q4" Oct 30 01:48:35.304351 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:35.304312 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2b8r\" (UniqueName: \"kubernetes.io/projected/d5b94b56-1785-45b7-ae04-3afb97ace8a3-kube-api-access-l2b8r\") pod \"maas-api-e2e-shared-b-w4-657fba-666fc89fb4-tc5q4\" (UID: \"d5b94b56-1785-45b7-ae04-3afb97ace8a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-657fba-666fc89fb4-tc5q4" Oct 30 01:48:35.323331 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:35.323303 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-657fba-666fc89fb4-tc5q4" Oct 30 01:48:35.625085 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:35.625062 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-657fba-666fc89fb4-tc5q4"] Oct 30 01:48:35.627308 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:48:35.627279 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5b94b56_1785_45b7_ae04_3afb97ace8a3.slice/crio-20b667b9456903413f99396eb274a1d94f7fd146cc42faa81980dba047e50346 WatchSource:0}: Error finding container 20b667b9456903413f99396eb274a1d94f7fd146cc42faa81980dba047e50346: Status 404 returned error can't find the container with id 20b667b9456903413f99396eb274a1d94f7fd146cc42faa81980dba047e50346 Oct 30 01:48:35.628835 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:35.628816 6221 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:48:36.305838 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:36.304566 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-657fba-666fc89fb4-tc5q4" event={"ID":"d5b94b56-1785-45b7-ae04-3afb97ace8a3","Type":"ContainerStarted","Data":"61ee90e9490812fe7a7868fe3df1b338024c735a94f871c1de3d4d94738736dd"} Oct 30 01:48:36.305838 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:36.304638 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-657fba-666fc89fb4-tc5q4" event={"ID":"d5b94b56-1785-45b7-ae04-3afb97ace8a3","Type":"ContainerStarted","Data":"20b667b9456903413f99396eb274a1d94f7fd146cc42faa81980dba047e50346"} Oct 30 01:48:36.305838 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:36.305720 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-657fba-666fc89fb4-tc5q4" Oct 30 01:48:36.385244 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:36.385192 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-657fba-666fc89fb4-tc5q4" podStartSLOduration=1.9679542730000001 podStartE2EDuration="2.385177946s" totalImagesPullingTime="417.223673ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:48:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:48:36.382410367 +0000 UTC m=+1839.960697724" watchObservedRunningTime="2026-10-30 01:48:36.385177946 +0000 UTC m=+1839.963465281" Oct 30 01:48:38.483331 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:38.483296 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:48:43.335848 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:43.335808 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-657fba-666fc89fb4-tc5q4" Oct 30 01:48:50.423736 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:50.423704 6221 generic.go:356] "Generic (PLEG): container finished" podID="6b685aa0-0c58-4911-8f31-3e920b8151a1" containerID="10902634aa9bb9a9f3db67ba6c2317fe28fee31b321eb17ddfaa4f4be3a78eb4" exitCode=0 Oct 30 01:48:50.424208 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:50.423745 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" event={"ID":"6b685aa0-0c58-4911-8f31-3e920b8151a1","Type":"ContainerDied","Data":"10902634aa9bb9a9f3db67ba6c2317fe28fee31b321eb17ddfaa4f4be3a78eb4"} Oct 30 01:48:50.424208 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:50.424112 6221 scope.go:122] "RemoveContainer" containerID="10902634aa9bb9a9f3db67ba6c2317fe28fee31b321eb17ddfaa4f4be3a78eb4" Oct 30 01:48:50.425663 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:50.425631 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:48:50.425788 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:50.425714 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:48:50.425788 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:50.425769 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:48:51.555670 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:51.554171 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" event={"ID":"6b685aa0-0c58-4911-8f31-3e920b8151a1","Type":"ContainerStarted","Data":"c164259cc14b3d0b048af1f8ee4f309f936ad612947e117f589ff7717b8d1c6b"} Oct 30 01:48:51.603172 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:48:51.603131 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:48:52.371313 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:52.371276 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" Oct 30 01:48:52.372962 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:52.372933 6221 patch_prober.go:29] interesting pod/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 01:48:52.373100 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:52.372995 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" podUID="6b685aa0-0c58-4911-8f31-3e920b8151a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 01:48:53.372109 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:53.372065 6221 patch_prober.go:29] interesting pod/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 01:48:53.372696 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:53.372121 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" podUID="6b685aa0-0c58-4911-8f31-3e920b8151a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 01:48:53.548593 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:53.545783 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" Oct 30 01:48:54.371410 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:54.371372 6221 patch_prober.go:29] interesting pod/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 01:48:54.371684 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:54.371426 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" podUID="6b685aa0-0c58-4911-8f31-3e920b8151a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 01:48:55.371750 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:55.371710 6221 patch_prober.go:29] interesting pod/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 01:48:55.372186 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:55.371767 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" podUID="6b685aa0-0c58-4911-8f31-3e920b8151a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 01:48:56.373882 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:56.373844 6221 patch_prober.go:29] interesting pod/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 01:48:56.374500 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:56.374460 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" podUID="6b685aa0-0c58-4911-8f31-3e920b8151a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 01:48:57.388358 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:57.388319 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" Oct 30 01:48:57.397062 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:48:57.397024 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" Oct 30 01:49:00.803232 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:00.803194 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-b6b7fc/auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp"] Oct 30 01:49:00.841610 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:00.838076 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-b6b7fc/auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp" Oct 30 01:49:00.841610 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:00.840389 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-b6b7fc/auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp"] Oct 30 01:49:00.856031 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:00.854591 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-b6b7fc\"/\"auth-test-model-b6b7fc-kserve-self-signed-certs\"" Oct 30 01:49:00.856031 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:00.854617 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-b6b7fc\"/\"default-dockercfg-znxdd\"" Oct 30 01:49:00.866955 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:00.866929 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-b6b7fc\"/\"openshift-service-ca.crt\"" Oct 30 01:49:00.867178 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:00.866977 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-b6b7fc\"/\"kube-root-ca.crt\"" Oct 30 01:49:00.885846 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:00.885815 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0794883c-76c6-4a6f-8f12-6e65420ef583-home\") pod \"auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp\" (UID: \"0794883c-76c6-4a6f-8f12-6e65420ef583\") " pod="ai-tenant-e2e-shared-a-w2-b6b7fc/auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp" Oct 30 01:49:00.886008 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:00.885862 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0794883c-76c6-4a6f-8f12-6e65420ef583-dshm\") pod \"auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp\" (UID: \"0794883c-76c6-4a6f-8f12-6e65420ef583\") " pod="ai-tenant-e2e-shared-a-w2-b6b7fc/auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp" Oct 30 01:49:00.886008 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:00.885934 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z2ks\" (UniqueName: \"kubernetes.io/projected/0794883c-76c6-4a6f-8f12-6e65420ef583-kube-api-access-7z2ks\") pod \"auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp\" (UID: \"0794883c-76c6-4a6f-8f12-6e65420ef583\") " pod="ai-tenant-e2e-shared-a-w2-b6b7fc/auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp" Oct 30 01:49:00.886008 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:00.886001 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0794883c-76c6-4a6f-8f12-6e65420ef583-tmp-dir\") pod \"auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp\" (UID: \"0794883c-76c6-4a6f-8f12-6e65420ef583\") " pod="ai-tenant-e2e-shared-a-w2-b6b7fc/auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp" Oct 30 01:49:00.886144 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:00.886047 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0794883c-76c6-4a6f-8f12-6e65420ef583-model-cache\") pod \"auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp\" (UID: \"0794883c-76c6-4a6f-8f12-6e65420ef583\") " pod="ai-tenant-e2e-shared-a-w2-b6b7fc/auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp" Oct 30 01:49:00.886144 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:00.886123 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0794883c-76c6-4a6f-8f12-6e65420ef583-tls-certs\") pod \"auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp\" (UID: \"0794883c-76c6-4a6f-8f12-6e65420ef583\") " pod="ai-tenant-e2e-shared-a-w2-b6b7fc/auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp" Oct 30 01:49:00.987248 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:00.987204 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0794883c-76c6-4a6f-8f12-6e65420ef583-tls-certs\") pod \"auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp\" (UID: \"0794883c-76c6-4a6f-8f12-6e65420ef583\") " pod="ai-tenant-e2e-shared-a-w2-b6b7fc/auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp" Oct 30 01:49:00.987434 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:00.987363 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0794883c-76c6-4a6f-8f12-6e65420ef583-home\") pod \"auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp\" (UID: \"0794883c-76c6-4a6f-8f12-6e65420ef583\") " pod="ai-tenant-e2e-shared-a-w2-b6b7fc/auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp" Oct 30 01:49:00.987434 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:00.987399 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0794883c-76c6-4a6f-8f12-6e65420ef583-dshm\") pod \"auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp\" (UID: \"0794883c-76c6-4a6f-8f12-6e65420ef583\") " pod="ai-tenant-e2e-shared-a-w2-b6b7fc/auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp" Oct 30 01:49:00.987434 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:00.987417 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z2ks\" (UniqueName: \"kubernetes.io/projected/0794883c-76c6-4a6f-8f12-6e65420ef583-kube-api-access-7z2ks\") pod \"auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp\" (UID: \"0794883c-76c6-4a6f-8f12-6e65420ef583\") " pod="ai-tenant-e2e-shared-a-w2-b6b7fc/auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp" Oct 30 01:49:00.987632 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:00.987476 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0794883c-76c6-4a6f-8f12-6e65420ef583-tmp-dir\") pod \"auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp\" (UID: \"0794883c-76c6-4a6f-8f12-6e65420ef583\") " pod="ai-tenant-e2e-shared-a-w2-b6b7fc/auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp" Oct 30 01:49:00.987632 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:00.987502 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0794883c-76c6-4a6f-8f12-6e65420ef583-model-cache\") pod \"auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp\" (UID: \"0794883c-76c6-4a6f-8f12-6e65420ef583\") " pod="ai-tenant-e2e-shared-a-w2-b6b7fc/auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp" Oct 30 01:49:00.987861 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:00.987834 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0794883c-76c6-4a6f-8f12-6e65420ef583-home\") pod \"auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp\" (UID: \"0794883c-76c6-4a6f-8f12-6e65420ef583\") " pod="ai-tenant-e2e-shared-a-w2-b6b7fc/auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp" Oct 30 01:49:00.987993 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:00.987879 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0794883c-76c6-4a6f-8f12-6e65420ef583-tmp-dir\") pod \"auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp\" (UID: \"0794883c-76c6-4a6f-8f12-6e65420ef583\") " pod="ai-tenant-e2e-shared-a-w2-b6b7fc/auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp" Oct 30 01:49:00.987993 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:00.987909 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0794883c-76c6-4a6f-8f12-6e65420ef583-model-cache\") pod \"auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp\" (UID: \"0794883c-76c6-4a6f-8f12-6e65420ef583\") " pod="ai-tenant-e2e-shared-a-w2-b6b7fc/auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp" Oct 30 01:49:00.989638 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:00.989611 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0794883c-76c6-4a6f-8f12-6e65420ef583-dshm\") pod \"auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp\" (UID: \"0794883c-76c6-4a6f-8f12-6e65420ef583\") " pod="ai-tenant-e2e-shared-a-w2-b6b7fc/auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp" Oct 30 01:49:00.989937 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:00.989919 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0794883c-76c6-4a6f-8f12-6e65420ef583-tls-certs\") pod \"auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp\" (UID: \"0794883c-76c6-4a6f-8f12-6e65420ef583\") " pod="ai-tenant-e2e-shared-a-w2-b6b7fc/auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp" Oct 30 01:49:01.027566 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:01.006899 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z2ks\" (UniqueName: \"kubernetes.io/projected/0794883c-76c6-4a6f-8f12-6e65420ef583-kube-api-access-7z2ks\") pod \"auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp\" (UID: \"0794883c-76c6-4a6f-8f12-6e65420ef583\") " pod="ai-tenant-e2e-shared-a-w2-b6b7fc/auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp" Oct 30 01:49:01.156417 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:01.156077 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-b6b7fc/auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp" Oct 30 01:49:01.541887 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:01.541858 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-b6b7fc/auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp"] Oct 30 01:49:01.543406 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:49:01.543385 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0794883c_76c6_4a6f_8f12_6e65420ef583.slice/crio-97cd20fcd98f07289ebfafbfa818effa2ada49e063321ca70da745aaa442f59d WatchSource:0}: Error finding container 97cd20fcd98f07289ebfafbfa818effa2ada49e063321ca70da745aaa442f59d: Status 404 returned error can't find the container with id 97cd20fcd98f07289ebfafbfa818effa2ada49e063321ca70da745aaa442f59d Oct 30 01:49:01.651738 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:01.651695 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-b6b7fc/auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp" event={"ID":"0794883c-76c6-4a6f-8f12-6e65420ef583","Type":"ContainerStarted","Data":"fd343432d731d1f3615c8f72c3e9cf0a270f19d6bf0552e7fd004ec57b35ba9a"} Oct 30 01:49:01.651920 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:01.651747 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-b6b7fc/auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp" event={"ID":"0794883c-76c6-4a6f-8f12-6e65420ef583","Type":"ContainerStarted","Data":"97cd20fcd98f07289ebfafbfa818effa2ada49e063321ca70da745aaa442f59d"} Oct 30 01:49:01.692901 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:01.692847 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-b6b7fc/auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp" podStartSLOduration=1.69280842 podStartE2EDuration="1.69280842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:49:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:49:01.690772756 +0000 UTC m=+1865.269060092" watchObservedRunningTime="2026-10-30 01:49:01.69280842 +0000 UTC m=+1865.271095755" Oct 30 01:49:02.157759 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:02.157716 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-b6b7fc/auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp" Oct 30 01:49:06.699821 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:06.699738 6221 generic.go:356] "Generic (PLEG): container finished" podID="56476948-dc26-41d2-9df4-e23f8a293d75" containerID="e19c9fa6d3979b03b5ef652ad07285786196e7158988e3dd4c87217b0087e747" exitCode=0 Oct 30 01:49:06.699821 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:06.699807 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" event={"ID":"56476948-dc26-41d2-9df4-e23f8a293d75","Type":"ContainerDied","Data":"e19c9fa6d3979b03b5ef652ad07285786196e7158988e3dd4c87217b0087e747"} Oct 30 01:49:06.700320 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:06.700229 6221 scope.go:122] "RemoveContainer" containerID="e19c9fa6d3979b03b5ef652ad07285786196e7158988e3dd4c87217b0087e747" Oct 30 01:49:06.701964 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:06.701937 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:49:06.702051 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:06.702006 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:49:06.702051 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:06.702038 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:49:07.709475 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:07.709442 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" event={"ID":"56476948-dc26-41d2-9df4-e23f8a293d75","Type":"ContainerStarted","Data":"77b25c8d03f17e9acf7d3fbe75a82ebf5d7a1a630fbd5420d9e588dcea5a5f01"} Oct 30 01:49:08.659373 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:08.659335 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" Oct 30 01:49:08.660717 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:08.660693 6221 patch_prober.go:29] interesting pod/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 01:49:08.660785 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:08.660740 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" podUID="56476948-dc26-41d2-9df4-e23f8a293d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 01:49:08.959410 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:08.959329 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" Oct 30 01:49:09.658979 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:09.658936 6221 patch_prober.go:29] interesting pod/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 01:49:09.659174 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:09.658999 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" podUID="56476948-dc26-41d2-9df4-e23f8a293d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 01:49:10.659096 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:10.659064 6221 patch_prober.go:29] interesting pod/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 01:49:10.659497 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:10.659127 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" podUID="56476948-dc26-41d2-9df4-e23f8a293d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 01:49:11.157374 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:11.157325 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-b6b7fc/auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp" Oct 30 01:49:11.170097 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:11.170068 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-b6b7fc/auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp" Oct 30 01:49:11.659850 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:11.659814 6221 patch_prober.go:29] interesting pod/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 01:49:11.660346 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:11.659879 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" podUID="56476948-dc26-41d2-9df4-e23f8a293d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 01:49:11.770041 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:11.770012 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-b6b7fc/auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp" Oct 30 01:49:12.659701 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:12.659658 6221 patch_prober.go:29] interesting pod/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 01:49:12.659878 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:12.659726 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" podUID="56476948-dc26-41d2-9df4-e23f8a293d75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 01:49:12.742723 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:12.742688 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:49:13.682817 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:13.682781 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" Oct 30 01:49:13.694227 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:13.694189 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" Oct 30 01:49:21.494942 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:49:21.494903 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:49:26.555365 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:26.555333 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-861cd2/auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5"] Oct 30 01:49:26.563216 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:26.563184 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-861cd2/auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5" Oct 30 01:49:26.572204 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:26.572173 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-861cd2/auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5"] Oct 30 01:49:26.579475 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:26.579448 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-861cd2\"/\"openshift-service-ca.crt\"" Oct 30 01:49:26.579639 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:26.579568 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-861cd2\"/\"kube-root-ca.crt\"" Oct 30 01:49:26.590572 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:26.590552 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-861cd2\"/\"auth-test-model-861cd2-kserve-self-signed-certs\"" Oct 30 01:49:26.590695 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:26.590611 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-861cd2\"/\"default-dockercfg-9bk8c\"" Oct 30 01:49:26.620340 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:26.620304 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d001b9fc-56db-4ba1-8fcc-1c1e3af02e12-tls-certs\") pod \"auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5\" (UID: \"d001b9fc-56db-4ba1-8fcc-1c1e3af02e12\") " pod="ai-tenant-e2e-shared-b-w1-861cd2/auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5" Oct 30 01:49:26.620511 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:26.620365 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d001b9fc-56db-4ba1-8fcc-1c1e3af02e12-dshm\") pod \"auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5\" (UID: \"d001b9fc-56db-4ba1-8fcc-1c1e3af02e12\") " pod="ai-tenant-e2e-shared-b-w1-861cd2/auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5" Oct 30 01:49:26.620608 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:26.620578 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgvd6\" (UniqueName: \"kubernetes.io/projected/d001b9fc-56db-4ba1-8fcc-1c1e3af02e12-kube-api-access-rgvd6\") pod \"auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5\" (UID: \"d001b9fc-56db-4ba1-8fcc-1c1e3af02e12\") " pod="ai-tenant-e2e-shared-b-w1-861cd2/auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5" Oct 30 01:49:26.620676 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:26.620610 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d001b9fc-56db-4ba1-8fcc-1c1e3af02e12-home\") pod \"auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5\" (UID: \"d001b9fc-56db-4ba1-8fcc-1c1e3af02e12\") " pod="ai-tenant-e2e-shared-b-w1-861cd2/auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5" Oct 30 01:49:26.620869 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:26.620854 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d001b9fc-56db-4ba1-8fcc-1c1e3af02e12-tmp-dir\") pod \"auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5\" (UID: \"d001b9fc-56db-4ba1-8fcc-1c1e3af02e12\") " pod="ai-tenant-e2e-shared-b-w1-861cd2/auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5" Oct 30 01:49:26.620925 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:26.620910 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d001b9fc-56db-4ba1-8fcc-1c1e3af02e12-model-cache\") pod \"auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5\" (UID: \"d001b9fc-56db-4ba1-8fcc-1c1e3af02e12\") " pod="ai-tenant-e2e-shared-b-w1-861cd2/auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5" Oct 30 01:49:26.722244 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:26.722214 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d001b9fc-56db-4ba1-8fcc-1c1e3af02e12-tmp-dir\") pod \"auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5\" (UID: \"d001b9fc-56db-4ba1-8fcc-1c1e3af02e12\") " pod="ai-tenant-e2e-shared-b-w1-861cd2/auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5" Oct 30 01:49:26.722446 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:26.722269 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d001b9fc-56db-4ba1-8fcc-1c1e3af02e12-model-cache\") pod \"auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5\" (UID: \"d001b9fc-56db-4ba1-8fcc-1c1e3af02e12\") " pod="ai-tenant-e2e-shared-b-w1-861cd2/auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5" Oct 30 01:49:26.722446 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:26.722305 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d001b9fc-56db-4ba1-8fcc-1c1e3af02e12-tls-certs\") pod \"auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5\" (UID: \"d001b9fc-56db-4ba1-8fcc-1c1e3af02e12\") " pod="ai-tenant-e2e-shared-b-w1-861cd2/auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5" Oct 30 01:49:26.722446 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:26.722333 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d001b9fc-56db-4ba1-8fcc-1c1e3af02e12-dshm\") pod \"auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5\" (UID: \"d001b9fc-56db-4ba1-8fcc-1c1e3af02e12\") " pod="ai-tenant-e2e-shared-b-w1-861cd2/auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5" Oct 30 01:49:26.722446 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:26.722431 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgvd6\" (UniqueName: \"kubernetes.io/projected/d001b9fc-56db-4ba1-8fcc-1c1e3af02e12-kube-api-access-rgvd6\") pod \"auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5\" (UID: \"d001b9fc-56db-4ba1-8fcc-1c1e3af02e12\") " pod="ai-tenant-e2e-shared-b-w1-861cd2/auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5" Oct 30 01:49:26.722720 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:26.722455 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d001b9fc-56db-4ba1-8fcc-1c1e3af02e12-home\") pod \"auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5\" (UID: \"d001b9fc-56db-4ba1-8fcc-1c1e3af02e12\") " pod="ai-tenant-e2e-shared-b-w1-861cd2/auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5" Oct 30 01:49:26.722720 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:26.722693 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d001b9fc-56db-4ba1-8fcc-1c1e3af02e12-tmp-dir\") pod \"auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5\" (UID: \"d001b9fc-56db-4ba1-8fcc-1c1e3af02e12\") " pod="ai-tenant-e2e-shared-b-w1-861cd2/auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5" Oct 30 01:49:26.722832 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:26.722738 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d001b9fc-56db-4ba1-8fcc-1c1e3af02e12-model-cache\") pod \"auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5\" (UID: \"d001b9fc-56db-4ba1-8fcc-1c1e3af02e12\") " pod="ai-tenant-e2e-shared-b-w1-861cd2/auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5" Oct 30 01:49:26.722949 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:26.722925 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d001b9fc-56db-4ba1-8fcc-1c1e3af02e12-home\") pod \"auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5\" (UID: \"d001b9fc-56db-4ba1-8fcc-1c1e3af02e12\") " pod="ai-tenant-e2e-shared-b-w1-861cd2/auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5" Oct 30 01:49:26.724888 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:26.724863 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d001b9fc-56db-4ba1-8fcc-1c1e3af02e12-dshm\") pod \"auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5\" (UID: \"d001b9fc-56db-4ba1-8fcc-1c1e3af02e12\") " pod="ai-tenant-e2e-shared-b-w1-861cd2/auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5" Oct 30 01:49:26.725033 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:26.725017 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d001b9fc-56db-4ba1-8fcc-1c1e3af02e12-tls-certs\") pod \"auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5\" (UID: \"d001b9fc-56db-4ba1-8fcc-1c1e3af02e12\") " pod="ai-tenant-e2e-shared-b-w1-861cd2/auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5" Oct 30 01:49:26.741556 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:26.741506 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgvd6\" (UniqueName: \"kubernetes.io/projected/d001b9fc-56db-4ba1-8fcc-1c1e3af02e12-kube-api-access-rgvd6\") pod \"auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5\" (UID: \"d001b9fc-56db-4ba1-8fcc-1c1e3af02e12\") " pod="ai-tenant-e2e-shared-b-w1-861cd2/auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5" Oct 30 01:49:26.885148 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:26.885058 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-861cd2/auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5" Oct 30 01:49:27.288693 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:27.288668 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-861cd2/auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5"] Oct 30 01:49:27.291072 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:49:27.291043 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd001b9fc_56db_4ba1_8fcc_1c1e3af02e12.slice/crio-5bb7a4be5d01bc1cca666d28fff2ed0a1b4519fcd2db330b7e3096f5bc5b1397 WatchSource:0}: Error finding container 5bb7a4be5d01bc1cca666d28fff2ed0a1b4519fcd2db330b7e3096f5bc5b1397: Status 404 returned error can't find the container with id 5bb7a4be5d01bc1cca666d28fff2ed0a1b4519fcd2db330b7e3096f5bc5b1397 Oct 30 01:49:27.923237 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:27.923201 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-861cd2/auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5" event={"ID":"d001b9fc-56db-4ba1-8fcc-1c1e3af02e12","Type":"ContainerStarted","Data":"cc66566cae7f2d999908d8c331e05e979b341771a1e5860023e1b18b94757724"} Oct 30 01:49:27.923695 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:27.923247 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-861cd2/auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5" event={"ID":"d001b9fc-56db-4ba1-8fcc-1c1e3af02e12","Type":"ContainerStarted","Data":"5bb7a4be5d01bc1cca666d28fff2ed0a1b4519fcd2db330b7e3096f5bc5b1397"} Oct 30 01:49:27.979579 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:27.979474 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-861cd2/auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5" podStartSLOduration=1.979454487 podStartE2EDuration="1.979454487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:49:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:49:27.973963679 +0000 UTC m=+1891.552251015" watchObservedRunningTime="2026-10-30 01:49:27.979454487 +0000 UTC m=+1891.557741826" Oct 30 01:49:28.886021 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:28.885989 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-861cd2/auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5" Oct 30 01:49:36.886083 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:36.886048 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-861cd2/auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5" Oct 30 01:49:36.898768 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:36.898739 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-861cd2/auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5" Oct 30 01:49:37.032227 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:37.032196 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-861cd2/auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5" Oct 30 01:49:46.565937 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:46.565853 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-389dba-99dbf7cdd-8wpwx"] Oct 30 01:49:51.495389 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:49:51.495344 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:49:51.566524 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:51.566448 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-389dba-99dbf7cdd-8wpwx" podUID="cbf9b943-008e-4887-abef-556d30970ff5" containerName="payload-processing" containerID="cri-o://d006451191c5b404351b3408076f0e593b4f17312d946cf7aa9d602791e1aafa" gracePeriod=25 Oct 30 01:49:51.862477 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:51.862450 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-389dba-99dbf7cdd-8wpwx" Oct 30 01:49:51.876973 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:51.876933 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbf9b943-008e-4887-abef-556d30970ff5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbf9b943-008e-4887-abef-556d30970ff5-plugins-config-volume\") pod \"cbf9b943-008e-4887-abef-556d30970ff5\" (UID: \"cbf9b943-008e-4887-abef-556d30970ff5\") " Oct 30 01:49:51.876973 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:51.876970 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbf9b943-008e-4887-abef-556d30970ff5-kube-api-access-wzdf6\" (UniqueName: \"kubernetes.io/projected/cbf9b943-008e-4887-abef-556d30970ff5-kube-api-access-wzdf6\") pod \"cbf9b943-008e-4887-abef-556d30970ff5\" (UID: \"cbf9b943-008e-4887-abef-556d30970ff5\") " Oct 30 01:49:51.877298 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:51.877275 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbf9b943-008e-4887-abef-556d30970ff5-plugins-config-volume" pod "cbf9b943-008e-4887-abef-556d30970ff5" (UID: "cbf9b943-008e-4887-abef-556d30970ff5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:49:51.877700 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:51.877681 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbf9b943-008e-4887-abef-556d30970ff5-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:49:51.879028 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:51.879008 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbf9b943-008e-4887-abef-556d30970ff5-kube-api-access-wzdf6" pod "cbf9b943-008e-4887-abef-556d30970ff5" (UID: "cbf9b943-008e-4887-abef-556d30970ff5"). InnerVolumeSpecName "kube-api-access-wzdf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:49:51.978980 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:51.978941 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzdf6\" (UniqueName: \"kubernetes.io/projected/cbf9b943-008e-4887-abef-556d30970ff5-kube-api-access-wzdf6\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:49:52.156656 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:52.156555 6221 generic.go:356] "Generic (PLEG): container finished" podID="cbf9b943-008e-4887-abef-556d30970ff5" containerID="d006451191c5b404351b3408076f0e593b4f17312d946cf7aa9d602791e1aafa" exitCode=0 Oct 30 01:49:52.156656 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:52.156643 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-389dba-99dbf7cdd-8wpwx" event={"ID":"cbf9b943-008e-4887-abef-556d30970ff5","Type":"ContainerDied","Data":"d006451191c5b404351b3408076f0e593b4f17312d946cf7aa9d602791e1aafa"} Oct 30 01:49:52.156656 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:52.156652 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-389dba-99dbf7cdd-8wpwx" Oct 30 01:49:52.156933 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:52.156709 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-389dba-99dbf7cdd-8wpwx" event={"ID":"cbf9b943-008e-4887-abef-556d30970ff5","Type":"ContainerDied","Data":"87c4892ccfce186cc5e593c8e8778c3a023f9966b2e46d6b67aa8864865a0f22"} Oct 30 01:49:52.156933 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:52.156731 6221 scope.go:122] "RemoveContainer" containerID="d006451191c5b404351b3408076f0e593b4f17312d946cf7aa9d602791e1aafa" Oct 30 01:49:52.167800 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:52.167782 6221 scope.go:122] "RemoveContainer" containerID="d006451191c5b404351b3408076f0e593b4f17312d946cf7aa9d602791e1aafa" Oct 30 01:49:52.168064 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:49:52.168044 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d006451191c5b404351b3408076f0e593b4f17312d946cf7aa9d602791e1aafa\": container with ID starting with d006451191c5b404351b3408076f0e593b4f17312d946cf7aa9d602791e1aafa not found: ID does not exist" containerID="d006451191c5b404351b3408076f0e593b4f17312d946cf7aa9d602791e1aafa" Oct 30 01:49:52.168137 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:52.168077 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d006451191c5b404351b3408076f0e593b4f17312d946cf7aa9d602791e1aafa"} err="failed to get container status \"d006451191c5b404351b3408076f0e593b4f17312d946cf7aa9d602791e1aafa\": rpc error: code = NotFound desc = could not find container \"d006451191c5b404351b3408076f0e593b4f17312d946cf7aa9d602791e1aafa\": container with ID starting with d006451191c5b404351b3408076f0e593b4f17312d946cf7aa9d602791e1aafa not found: ID does not exist" Oct 30 01:49:52.209881 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:52.209851 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-389dba-99dbf7cdd-8wpwx"] Oct 30 01:49:52.213015 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:52.212990 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-389dba-99dbf7cdd-8wpwx"] Oct 30 01:49:52.548236 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:49:52.548195 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbf9b943-008e-4887-abef-556d30970ff5" path="/var/lib/kubelet/pods/cbf9b943-008e-4887-abef-556d30970ff5/volumes" Oct 30 01:50:09.368297 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:50:09.368260 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:50:21.494745 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:50:21.494701 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:50:51.495048 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:50:51.495012 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:51:01.849018 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:51:01.848980 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:51:21.495459 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:51:21.495373 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:51:37.996581 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:51:37.994936 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:51:51.496121 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:51:51.496078 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:51:59.208460 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:51:59.208414 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c8d376/auth-test-model-c8d376-kserve-6bddf577bf-jw4k8"] Oct 30 01:51:59.491604 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:51:59.491474 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7486-ffa1d7"] Oct 30 01:51:59.492907 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:51:59.492864 6221 cpu_manager.go:414] "containerMap: removing container" podUID="cbf9b943-008e-4887-abef-556d30970ff5" containerName="payload-processing" Oct 30 01:51:59.493137 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:51:59.493124 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbf9b943-008e-4887-abef-556d30970ff5" containerName="payload-processing" Oct 30 01:51:59.493518 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:51:59.493497 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbf9b943-008e-4887-abef-556d30970ff5" containerName="payload-processing" Oct 30 01:51:59.498692 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:51:59.497345 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7486-ffa1d7" Oct 30 01:51:59.502100 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:51:59.502077 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7486-ffa1d7"] Oct 30 01:51:59.512997 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:51:59.512946 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pm9kh\"" Oct 30 01:51:59.516619 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:51:59.516248 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2nwz\" (UniqueName: \"kubernetes.io/projected/d2dedf6c-cc74-490e-b2cf-c79dd8986b28-kube-api-access-n2nwz\") pod \"test-curl-7486-ffa1d7\" (UID: \"d2dedf6c-cc74-490e-b2cf-c79dd8986b28\") " pod="opendatahub/test-curl-7486-ffa1d7" Oct 30 01:51:59.618290 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:51:59.618237 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2nwz\" (UniqueName: \"kubernetes.io/projected/d2dedf6c-cc74-490e-b2cf-c79dd8986b28-kube-api-access-n2nwz\") pod \"test-curl-7486-ffa1d7\" (UID: \"d2dedf6c-cc74-490e-b2cf-c79dd8986b28\") " pod="opendatahub/test-curl-7486-ffa1d7" Oct 30 01:51:59.639867 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:51:59.639829 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2nwz\" (UniqueName: \"kubernetes.io/projected/d2dedf6c-cc74-490e-b2cf-c79dd8986b28-kube-api-access-n2nwz\") pod \"test-curl-7486-ffa1d7\" (UID: \"d2dedf6c-cc74-490e-b2cf-c79dd8986b28\") " pod="opendatahub/test-curl-7486-ffa1d7" Oct 30 01:51:59.819496 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:51:59.819460 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7486-ffa1d7" Oct 30 01:52:00.219312 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:00.219287 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7486-ffa1d7"] Oct 30 01:52:00.221259 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:52:00.221231 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2dedf6c_cc74_490e_b2cf_c79dd8986b28.slice/crio-91bfc34e60c194f62bf7ab85240b56d00ad37f4b15288889190702b249cbd8f3 WatchSource:0}: Error finding container 91bfc34e60c194f62bf7ab85240b56d00ad37f4b15288889190702b249cbd8f3: Status 404 returned error can't find the container with id 91bfc34e60c194f62bf7ab85240b56d00ad37f4b15288889190702b249cbd8f3 Oct 30 01:52:00.403831 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:00.403791 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7486-ffa1d7" event={"ID":"d2dedf6c-cc74-490e-b2cf-c79dd8986b28","Type":"ContainerStarted","Data":"91bfc34e60c194f62bf7ab85240b56d00ad37f4b15288889190702b249cbd8f3"} Oct 30 01:52:03.427787 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:03.427746 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7486-ffa1d7" event={"ID":"d2dedf6c-cc74-490e-b2cf-c79dd8986b28","Type":"ContainerStarted","Data":"8bebcfedded15f5391c0e351f3e05b3180bb437cb42db05d1c84b811a09b182d"} Oct 30 01:52:03.466498 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:03.466440 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7486-ffa1d7" podStartSLOduration=2.211534782 podStartE2EDuration="4.466424546s" totalImagesPullingTime="2.254889764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:51:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:52:03.464326994 +0000 UTC m=+2047.042614329" watchObservedRunningTime="2026-10-30 01:52:03.466424546 +0000 UTC m=+2047.044711929" Oct 30 01:52:04.273489 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:04.273447 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7486-ffa1d7"] Oct 30 01:52:04.278835 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:04.278796 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7486-ffa1d7"] Oct 30 01:52:05.444971 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:05.444887 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7486-ffa1d7" podUID="d2dedf6c-cc74-490e-b2cf-c79dd8986b28" containerName="test-curl-7486-ffa1d7" containerID="cri-o://8bebcfedded15f5391c0e351f3e05b3180bb437cb42db05d1c84b811a09b182d" gracePeriod=2 Oct 30 01:52:07.836957 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:07.836925 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7486-ffa1d7" Oct 30 01:52:07.907493 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:07.907446 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2dedf6c-cc74-490e-b2cf-c79dd8986b28-kube-api-access-n2nwz\" (UniqueName: \"kubernetes.io/projected/d2dedf6c-cc74-490e-b2cf-c79dd8986b28-kube-api-access-n2nwz\") pod \"d2dedf6c-cc74-490e-b2cf-c79dd8986b28\" (UID: \"d2dedf6c-cc74-490e-b2cf-c79dd8986b28\") " Oct 30 01:52:07.909697 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:07.909666 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2dedf6c-cc74-490e-b2cf-c79dd8986b28-kube-api-access-n2nwz" pod "d2dedf6c-cc74-490e-b2cf-c79dd8986b28" (UID: "d2dedf6c-cc74-490e-b2cf-c79dd8986b28"). InnerVolumeSpecName "kube-api-access-n2nwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:52:08.009685 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:08.009589 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2nwz\" (UniqueName: \"kubernetes.io/projected/d2dedf6c-cc74-490e-b2cf-c79dd8986b28-kube-api-access-n2nwz\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:08.471030 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:08.470992 6221 generic.go:356] "Generic (PLEG): container finished" podID="d2dedf6c-cc74-490e-b2cf-c79dd8986b28" containerID="8bebcfedded15f5391c0e351f3e05b3180bb437cb42db05d1c84b811a09b182d" exitCode=137 Oct 30 01:52:08.471255 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:08.471038 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7486-ffa1d7" Oct 30 01:52:08.471255 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:08.471091 6221 scope.go:122] "RemoveContainer" containerID="8bebcfedded15f5391c0e351f3e05b3180bb437cb42db05d1c84b811a09b182d" Oct 30 01:52:08.482769 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:08.482747 6221 scope.go:122] "RemoveContainer" containerID="8bebcfedded15f5391c0e351f3e05b3180bb437cb42db05d1c84b811a09b182d" Oct 30 01:52:08.483077 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:52:08.483056 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bebcfedded15f5391c0e351f3e05b3180bb437cb42db05d1c84b811a09b182d\": container with ID starting with 8bebcfedded15f5391c0e351f3e05b3180bb437cb42db05d1c84b811a09b182d not found: ID does not exist" containerID="8bebcfedded15f5391c0e351f3e05b3180bb437cb42db05d1c84b811a09b182d" Oct 30 01:52:08.483128 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:08.483088 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bebcfedded15f5391c0e351f3e05b3180bb437cb42db05d1c84b811a09b182d"} err="failed to get container status \"8bebcfedded15f5391c0e351f3e05b3180bb437cb42db05d1c84b811a09b182d\": rpc error: code = NotFound desc = could not find container \"8bebcfedded15f5391c0e351f3e05b3180bb437cb42db05d1c84b811a09b182d\": container with ID starting with 8bebcfedded15f5391c0e351f3e05b3180bb437cb42db05d1c84b811a09b182d not found: ID does not exist" Oct 30 01:52:08.494436 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:52:08.494400 6221 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7486-ffa1d7\" is forbidden: User \"system:node:ip-10-0-135-65.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-65.ec2.internal' and this object" podUID="d2dedf6c-cc74-490e-b2cf-c79dd8986b28" pod="opendatahub/test-curl-7486-ffa1d7" Oct 30 01:52:08.548555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:08.548509 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2dedf6c-cc74-490e-b2cf-c79dd8986b28" path="/var/lib/kubelet/pods/d2dedf6c-cc74-490e-b2cf-c79dd8986b28/volumes" Oct 30 01:52:14.232342 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:14.232299 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-c8d376/auth-test-model-c8d376-kserve-6bddf577bf-jw4k8" podUID="2a8dd470-155e-4470-b5e5-01ee5c9a0065" containerName="main" containerID="cri-o://313066bfe22113b498f4b83cebbe2e9f05f5ab1ca17c4b5297206a6f65ec8344" gracePeriod=45 Oct 30 01:52:14.506681 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:14.506651 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-c8d376/auth-test-model-c8d376-kserve-6bddf577bf-jw4k8" Oct 30 01:52:14.523140 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:14.523104 6221 generic.go:356] "Generic (PLEG): container finished" podID="2a8dd470-155e-4470-b5e5-01ee5c9a0065" containerID="313066bfe22113b498f4b83cebbe2e9f05f5ab1ca17c4b5297206a6f65ec8344" exitCode=0 Oct 30 01:52:14.523327 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:14.523166 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-c8d376/auth-test-model-c8d376-kserve-6bddf577bf-jw4k8" Oct 30 01:52:14.523327 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:14.523185 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-c8d376/auth-test-model-c8d376-kserve-6bddf577bf-jw4k8" event={"ID":"2a8dd470-155e-4470-b5e5-01ee5c9a0065","Type":"ContainerDied","Data":"313066bfe22113b498f4b83cebbe2e9f05f5ab1ca17c4b5297206a6f65ec8344"} Oct 30 01:52:14.523327 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:14.523241 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-c8d376/auth-test-model-c8d376-kserve-6bddf577bf-jw4k8" event={"ID":"2a8dd470-155e-4470-b5e5-01ee5c9a0065","Type":"ContainerDied","Data":"0fb2407e7530decd2032dfaee6bdcadc2e8b7dcea0742728766c5017410dd362"} Oct 30 01:52:14.523327 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:14.523264 6221 scope.go:122] "RemoveContainer" containerID="313066bfe22113b498f4b83cebbe2e9f05f5ab1ca17c4b5297206a6f65ec8344" Oct 30 01:52:14.614891 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:14.614861 6221 scope.go:122] "RemoveContainer" containerID="313066bfe22113b498f4b83cebbe2e9f05f5ab1ca17c4b5297206a6f65ec8344" Oct 30 01:52:14.615284 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:52:14.615259 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"313066bfe22113b498f4b83cebbe2e9f05f5ab1ca17c4b5297206a6f65ec8344\": container with ID starting with 313066bfe22113b498f4b83cebbe2e9f05f5ab1ca17c4b5297206a6f65ec8344 not found: ID does not exist" containerID="313066bfe22113b498f4b83cebbe2e9f05f5ab1ca17c4b5297206a6f65ec8344" Oct 30 01:52:14.615406 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:14.615297 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"313066bfe22113b498f4b83cebbe2e9f05f5ab1ca17c4b5297206a6f65ec8344"} err="failed to get container status \"313066bfe22113b498f4b83cebbe2e9f05f5ab1ca17c4b5297206a6f65ec8344\": rpc error: code = NotFound desc = could not find container \"313066bfe22113b498f4b83cebbe2e9f05f5ab1ca17c4b5297206a6f65ec8344\": container with ID starting with 313066bfe22113b498f4b83cebbe2e9f05f5ab1ca17c4b5297206a6f65ec8344 not found: ID does not exist" Oct 30 01:52:14.679478 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:14.679447 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a8dd470-155e-4470-b5e5-01ee5c9a0065-tls-certs\" (UniqueName: \"kubernetes.io/secret/2a8dd470-155e-4470-b5e5-01ee5c9a0065-tls-certs\") pod \"2a8dd470-155e-4470-b5e5-01ee5c9a0065\" (UID: \"2a8dd470-155e-4470-b5e5-01ee5c9a0065\") " Oct 30 01:52:14.679716 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:14.679696 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a8dd470-155e-4470-b5e5-01ee5c9a0065-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a8dd470-155e-4470-b5e5-01ee5c9a0065-model-cache\") pod \"2a8dd470-155e-4470-b5e5-01ee5c9a0065\" (UID: \"2a8dd470-155e-4470-b5e5-01ee5c9a0065\") " Oct 30 01:52:14.679813 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:14.679799 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a8dd470-155e-4470-b5e5-01ee5c9a0065-home\" (UniqueName: \"kubernetes.io/empty-dir/2a8dd470-155e-4470-b5e5-01ee5c9a0065-home\") pod \"2a8dd470-155e-4470-b5e5-01ee5c9a0065\" (UID: \"2a8dd470-155e-4470-b5e5-01ee5c9a0065\") " Oct 30 01:52:14.679869 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:14.679841 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a8dd470-155e-4470-b5e5-01ee5c9a0065-kube-api-access-7l6b5\" (UniqueName: \"kubernetes.io/projected/2a8dd470-155e-4470-b5e5-01ee5c9a0065-kube-api-access-7l6b5\") pod \"2a8dd470-155e-4470-b5e5-01ee5c9a0065\" (UID: \"2a8dd470-155e-4470-b5e5-01ee5c9a0065\") " Oct 30 01:52:14.679933 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:14.679906 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a8dd470-155e-4470-b5e5-01ee5c9a0065-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a8dd470-155e-4470-b5e5-01ee5c9a0065-dshm\") pod \"2a8dd470-155e-4470-b5e5-01ee5c9a0065\" (UID: \"2a8dd470-155e-4470-b5e5-01ee5c9a0065\") " Oct 30 01:52:14.679987 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:14.679941 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a8dd470-155e-4470-b5e5-01ee5c9a0065-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a8dd470-155e-4470-b5e5-01ee5c9a0065-tmp-dir\") pod \"2a8dd470-155e-4470-b5e5-01ee5c9a0065\" (UID: \"2a8dd470-155e-4470-b5e5-01ee5c9a0065\") " Oct 30 01:52:14.679987 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:14.679951 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a8dd470-155e-4470-b5e5-01ee5c9a0065-model-cache" pod "2a8dd470-155e-4470-b5e5-01ee5c9a0065" (UID: "2a8dd470-155e-4470-b5e5-01ee5c9a0065"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:14.680089 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:14.680047 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a8dd470-155e-4470-b5e5-01ee5c9a0065-home" pod "2a8dd470-155e-4470-b5e5-01ee5c9a0065" (UID: "2a8dd470-155e-4470-b5e5-01ee5c9a0065"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:14.680227 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:14.680206 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a8dd470-155e-4470-b5e5-01ee5c9a0065-tmp-dir" pod "2a8dd470-155e-4470-b5e5-01ee5c9a0065" (UID: "2a8dd470-155e-4470-b5e5-01ee5c9a0065"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:14.680740 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:14.680719 6221 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a8dd470-155e-4470-b5e5-01ee5c9a0065-model-cache\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:14.680879 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:14.680856 6221 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a8dd470-155e-4470-b5e5-01ee5c9a0065-home\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:14.680879 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:14.680873 6221 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a8dd470-155e-4470-b5e5-01ee5c9a0065-tmp-dir\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:14.681882 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:14.681858 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a8dd470-155e-4470-b5e5-01ee5c9a0065-tls-certs" pod "2a8dd470-155e-4470-b5e5-01ee5c9a0065" (UID: "2a8dd470-155e-4470-b5e5-01ee5c9a0065"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:52:14.681978 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:14.681962 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a8dd470-155e-4470-b5e5-01ee5c9a0065-dshm" pod "2a8dd470-155e-4470-b5e5-01ee5c9a0065" (UID: "2a8dd470-155e-4470-b5e5-01ee5c9a0065"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:14.682070 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:14.682051 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a8dd470-155e-4470-b5e5-01ee5c9a0065-kube-api-access-7l6b5" pod "2a8dd470-155e-4470-b5e5-01ee5c9a0065" (UID: "2a8dd470-155e-4470-b5e5-01ee5c9a0065"). InnerVolumeSpecName "kube-api-access-7l6b5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:52:14.781826 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:14.781802 6221 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a8dd470-155e-4470-b5e5-01ee5c9a0065-tls-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:14.781826 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:14.781827 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l6b5\" (UniqueName: \"kubernetes.io/projected/2a8dd470-155e-4470-b5e5-01ee5c9a0065-kube-api-access-7l6b5\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:14.782019 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:14.781838 6221 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a8dd470-155e-4470-b5e5-01ee5c9a0065-dshm\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:14.869490 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:14.869453 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c8d376/auth-test-model-c8d376-kserve-6bddf577bf-jw4k8"] Oct 30 01:52:14.872746 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:14.872727 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c8d376/auth-test-model-c8d376-kserve-6bddf577bf-jw4k8"] Oct 30 01:52:16.548375 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:16.548342 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a8dd470-155e-4470-b5e5-01ee5c9a0065" path="/var/lib/kubelet/pods/2a8dd470-155e-4470-b5e5-01ee5c9a0065/volumes" Oct 30 01:52:21.494725 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:52:21.494689 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:52:21.538020 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:21.537984 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n"] Oct 30 01:52:21.538263 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:21.538237 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n" podUID="39e768cb-a319-4661-8e6d-aa7d654bba94" containerName="istio-proxy" containerID="cri-o://fa6e37df39057bb97547c3f2ee79ac08ee8c7052a9c8d016c749718149cac5d3" gracePeriod=30 Oct 30 01:52:26.810144 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:26.810070 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n" Oct 30 01:52:26.866252 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:26.866219 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39e768cb-a319-4661-8e6d-aa7d654bba94-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39e768cb-a319-4661-8e6d-aa7d654bba94-istiod-ca-cert\") pod \"39e768cb-a319-4661-8e6d-aa7d654bba94\" (UID: \"39e768cb-a319-4661-8e6d-aa7d654bba94\") " Oct 30 01:52:26.866436 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:26.866271 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39e768cb-a319-4661-8e6d-aa7d654bba94-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39e768cb-a319-4661-8e6d-aa7d654bba94-istio-data\") pod \"39e768cb-a319-4661-8e6d-aa7d654bba94\" (UID: \"39e768cb-a319-4661-8e6d-aa7d654bba94\") " Oct 30 01:52:26.866436 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:26.866296 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39e768cb-a319-4661-8e6d-aa7d654bba94-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39e768cb-a319-4661-8e6d-aa7d654bba94-credential-socket\") pod \"39e768cb-a319-4661-8e6d-aa7d654bba94\" (UID: \"39e768cb-a319-4661-8e6d-aa7d654bba94\") " Oct 30 01:52:26.866436 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:26.866365 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39e768cb-a319-4661-8e6d-aa7d654bba94-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39e768cb-a319-4661-8e6d-aa7d654bba94-istio-envoy\") pod \"39e768cb-a319-4661-8e6d-aa7d654bba94\" (UID: \"39e768cb-a319-4661-8e6d-aa7d654bba94\") " Oct 30 01:52:26.866436 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:26.866411 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39e768cb-a319-4661-8e6d-aa7d654bba94-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39e768cb-a319-4661-8e6d-aa7d654bba94-istio-podinfo\") pod \"39e768cb-a319-4661-8e6d-aa7d654bba94\" (UID: \"39e768cb-a319-4661-8e6d-aa7d654bba94\") " Oct 30 01:52:26.866706 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:26.866456 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39e768cb-a319-4661-8e6d-aa7d654bba94-istio-token\" (UniqueName: \"kubernetes.io/projected/39e768cb-a319-4661-8e6d-aa7d654bba94-istio-token\") pod \"39e768cb-a319-4661-8e6d-aa7d654bba94\" (UID: \"39e768cb-a319-4661-8e6d-aa7d654bba94\") " Oct 30 01:52:26.866706 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:26.866602 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39e768cb-a319-4661-8e6d-aa7d654bba94-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39e768cb-a319-4661-8e6d-aa7d654bba94-workload-socket\") pod \"39e768cb-a319-4661-8e6d-aa7d654bba94\" (UID: \"39e768cb-a319-4661-8e6d-aa7d654bba94\") " Oct 30 01:52:26.866706 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:26.866657 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39e768cb-a319-4661-8e6d-aa7d654bba94-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39e768cb-a319-4661-8e6d-aa7d654bba94-workload-certs\") pod \"39e768cb-a319-4661-8e6d-aa7d654bba94\" (UID: \"39e768cb-a319-4661-8e6d-aa7d654bba94\") " Oct 30 01:52:26.866706 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:26.866673 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39e768cb-a319-4661-8e6d-aa7d654bba94-istio-data" pod "39e768cb-a319-4661-8e6d-aa7d654bba94" (UID: "39e768cb-a319-4661-8e6d-aa7d654bba94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:26.867147 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:26.866704 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39e768cb-a319-4661-8e6d-aa7d654bba94-istiod-ca-cert" pod "39e768cb-a319-4661-8e6d-aa7d654bba94" (UID: "39e768cb-a319-4661-8e6d-aa7d654bba94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:52:26.867147 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:26.866708 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39e768cb-a319-4661-8e6d-aa7d654bba94-credential-socket" pod "39e768cb-a319-4661-8e6d-aa7d654bba94" (UID: "39e768cb-a319-4661-8e6d-aa7d654bba94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:26.867147 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:26.866724 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39e768cb-a319-4661-8e6d-aa7d654bba94-kube-api-access-8l9qt\" (UniqueName: \"kubernetes.io/projected/39e768cb-a319-4661-8e6d-aa7d654bba94-kube-api-access-8l9qt\") pod \"39e768cb-a319-4661-8e6d-aa7d654bba94\" (UID: \"39e768cb-a319-4661-8e6d-aa7d654bba94\") " Oct 30 01:52:26.870493 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:26.866960 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39e768cb-a319-4661-8e6d-aa7d654bba94-workload-socket" pod "39e768cb-a319-4661-8e6d-aa7d654bba94" (UID: "39e768cb-a319-4661-8e6d-aa7d654bba94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:26.870701 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:26.867055 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39e768cb-a319-4661-8e6d-aa7d654bba94-workload-certs" pod "39e768cb-a319-4661-8e6d-aa7d654bba94" (UID: "39e768cb-a319-4661-8e6d-aa7d654bba94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:26.870701 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:26.867928 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39e768cb-a319-4661-8e6d-aa7d654bba94-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:26.870701 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:26.870549 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39e768cb-a319-4661-8e6d-aa7d654bba94-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:26.870701 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:26.870567 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39e768cb-a319-4661-8e6d-aa7d654bba94-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:26.871055 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:26.870730 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39e768cb-a319-4661-8e6d-aa7d654bba94-istio-podinfo" pod "39e768cb-a319-4661-8e6d-aa7d654bba94" (UID: "39e768cb-a319-4661-8e6d-aa7d654bba94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:52:26.871114 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:26.871102 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39e768cb-a319-4661-8e6d-aa7d654bba94-istio-envoy" pod "39e768cb-a319-4661-8e6d-aa7d654bba94" (UID: "39e768cb-a319-4661-8e6d-aa7d654bba94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:26.872251 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:26.872226 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39e768cb-a319-4661-8e6d-aa7d654bba94-kube-api-access-8l9qt" pod "39e768cb-a319-4661-8e6d-aa7d654bba94" (UID: "39e768cb-a319-4661-8e6d-aa7d654bba94"). InnerVolumeSpecName "kube-api-access-8l9qt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:52:26.873679 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:26.873657 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39e768cb-a319-4661-8e6d-aa7d654bba94-istio-token" pod "39e768cb-a319-4661-8e6d-aa7d654bba94" (UID: "39e768cb-a319-4661-8e6d-aa7d654bba94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:52:26.972013 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:26.971981 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39e768cb-a319-4661-8e6d-aa7d654bba94-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:26.972013 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:26.972010 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39e768cb-a319-4661-8e6d-aa7d654bba94-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:26.972013 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:26.972019 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39e768cb-a319-4661-8e6d-aa7d654bba94-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:26.972265 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:26.972027 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39e768cb-a319-4661-8e6d-aa7d654bba94-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:26.972265 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:26.972036 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39e768cb-a319-4661-8e6d-aa7d654bba94-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:26.972265 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:26.972045 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l9qt\" (UniqueName: \"kubernetes.io/projected/39e768cb-a319-4661-8e6d-aa7d654bba94-kube-api-access-8l9qt\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:27.626332 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:27.626299 6221 generic.go:356] "Generic (PLEG): container finished" podID="39e768cb-a319-4661-8e6d-aa7d654bba94" containerID="fa6e37df39057bb97547c3f2ee79ac08ee8c7052a9c8d016c749718149cac5d3" exitCode=0 Oct 30 01:52:27.626570 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:27.626359 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n" Oct 30 01:52:27.626570 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:27.626367 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n" event={"ID":"39e768cb-a319-4661-8e6d-aa7d654bba94","Type":"ContainerDied","Data":"fa6e37df39057bb97547c3f2ee79ac08ee8c7052a9c8d016c749718149cac5d3"} Oct 30 01:52:27.626570 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:27.626474 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n" event={"ID":"39e768cb-a319-4661-8e6d-aa7d654bba94","Type":"ContainerDied","Data":"ce0abdf73f1fc6f88036ffb2ea80ddbafd4de1c3afccb1ff8bb38d265ff2476a"} Oct 30 01:52:27.626570 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:27.626496 6221 scope.go:122] "RemoveContainer" containerID="fa6e37df39057bb97547c3f2ee79ac08ee8c7052a9c8d016c749718149cac5d3" Oct 30 01:52:27.637318 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:27.637297 6221 scope.go:122] "RemoveContainer" containerID="fa6e37df39057bb97547c3f2ee79ac08ee8c7052a9c8d016c749718149cac5d3" Oct 30 01:52:27.637592 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:52:27.637572 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa6e37df39057bb97547c3f2ee79ac08ee8c7052a9c8d016c749718149cac5d3\": container with ID starting with fa6e37df39057bb97547c3f2ee79ac08ee8c7052a9c8d016c749718149cac5d3 not found: ID does not exist" containerID="fa6e37df39057bb97547c3f2ee79ac08ee8c7052a9c8d016c749718149cac5d3" Oct 30 01:52:27.637640 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:27.637601 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa6e37df39057bb97547c3f2ee79ac08ee8c7052a9c8d016c749718149cac5d3"} err="failed to get container status \"fa6e37df39057bb97547c3f2ee79ac08ee8c7052a9c8d016c749718149cac5d3\": rpc error: code = NotFound desc = could not find container \"fa6e37df39057bb97547c3f2ee79ac08ee8c7052a9c8d016c749718149cac5d3\": container with ID starting with fa6e37df39057bb97547c3f2ee79ac08ee8c7052a9c8d016c749718149cac5d3 not found: ID does not exist" Oct 30 01:52:27.684093 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:27.684064 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n"] Oct 30 01:52:27.688081 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:27.688060 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-c225n"] Oct 30 01:52:28.549546 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:28.549504 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39e768cb-a319-4661-8e6d-aa7d654bba94" path="/var/lib/kubelet/pods/39e768cb-a319-4661-8e6d-aa7d654bba94/volumes" Oct 30 01:52:29.049074 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:29.049034 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:52:30.459646 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:30.459614 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr"] Oct 30 01:52:30.460107 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:30.459812 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr" podUID="40e8c5af-0173-4388-b938-ddc9b82ea112" containerName="istio-proxy" containerID="cri-o://ebef54a515eebaf98f3f0719c4d2c943c8b073037d929cad5d3f542928cc3c48" gracePeriod=30 Oct 30 01:52:32.216758 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:32.216714 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4f7628-7fd5f9b886-bgc5n"] Oct 30 01:52:32.217584 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:32.217562 6221 cpu_manager.go:414] "containerMap: removing container" podUID="2a8dd470-155e-4470-b5e5-01ee5c9a0065" containerName="main" Oct 30 01:52:32.217712 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:32.217584 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a8dd470-155e-4470-b5e5-01ee5c9a0065" containerName="main" Oct 30 01:52:32.217712 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:32.217606 6221 cpu_manager.go:414] "containerMap: removing container" podUID="39e768cb-a319-4661-8e6d-aa7d654bba94" containerName="istio-proxy" Oct 30 01:52:32.217712 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:32.217613 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39e768cb-a319-4661-8e6d-aa7d654bba94" containerName="istio-proxy" Oct 30 01:52:32.217712 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:32.217670 6221 cpu_manager.go:414] "containerMap: removing container" podUID="d2dedf6c-cc74-490e-b2cf-c79dd8986b28" containerName="test-curl-7486-ffa1d7" Oct 30 01:52:32.217712 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:32.217678 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2dedf6c-cc74-490e-b2cf-c79dd8986b28" containerName="test-curl-7486-ffa1d7" Oct 30 01:52:32.217968 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:32.217844 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a8dd470-155e-4470-b5e5-01ee5c9a0065" containerName="main" Oct 30 01:52:32.217968 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:32.217859 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="39e768cb-a319-4661-8e6d-aa7d654bba94" containerName="istio-proxy" Oct 30 01:52:32.217968 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:32.217879 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2dedf6c-cc74-490e-b2cf-c79dd8986b28" containerName="test-curl-7486-ffa1d7" Oct 30 01:52:32.223600 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:32.223577 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4f7628-7fd5f9b886-bgc5n" Oct 30 01:52:32.237593 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:32.237565 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4f7628-7fd5f9b886-bgc5n"] Oct 30 01:52:32.246496 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:32.246466 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-4f7628\"" Oct 30 01:52:32.276647 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:32.276608 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4f7628-5d4595b5c5-q9f88"] Oct 30 01:52:32.281948 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:32.281919 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4f7628-5d4595b5c5-q9f88" Oct 30 01:52:32.297334 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:32.297303 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-4f7628-dockercfg-kbqlb\"" Oct 30 01:52:32.297943 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:32.297913 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4f7628-5d4595b5c5-q9f88"] Oct 30 01:52:32.302095 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:32.302068 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbl4r\" (UniqueName: \"kubernetes.io/projected/7f16168b-61d0-4a08-94b7-7ea5581e4005-kube-api-access-nbl4r\") pod \"payload-pre-processing-e2e-shared-b-w3-4f7628-5d4595b5c5-q9f88\" (UID: \"7f16168b-61d0-4a08-94b7-7ea5581e4005\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4f7628-5d4595b5c5-q9f88" Oct 30 01:52:32.302210 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:32.302197 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f16168b-61d0-4a08-94b7-7ea5581e4005-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-4f7628-5d4595b5c5-q9f88\" (UID: \"7f16168b-61d0-4a08-94b7-7ea5581e4005\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4f7628-5d4595b5c5-q9f88" Oct 30 01:52:32.302297 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:32.302284 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ks9s\" (UniqueName: \"kubernetes.io/projected/a7f33e22-ce25-46bc-8add-b9aa4434e0e3-kube-api-access-7ks9s\") pod \"payload-processing-e2e-shared-b-w3-4f7628-7fd5f9b886-bgc5n\" (UID: \"a7f33e22-ce25-46bc-8add-b9aa4434e0e3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4f7628-7fd5f9b886-bgc5n" Oct 30 01:52:32.302348 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:32.302314 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7f33e22-ce25-46bc-8add-b9aa4434e0e3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-4f7628-7fd5f9b886-bgc5n\" (UID: \"a7f33e22-ce25-46bc-8add-b9aa4434e0e3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4f7628-7fd5f9b886-bgc5n" Oct 30 01:52:32.473606 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:32.472932 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f16168b-61d0-4a08-94b7-7ea5581e4005-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-4f7628-5d4595b5c5-q9f88\" (UID: \"7f16168b-61d0-4a08-94b7-7ea5581e4005\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4f7628-5d4595b5c5-q9f88" Oct 30 01:52:32.473606 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:32.473079 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ks9s\" (UniqueName: \"kubernetes.io/projected/a7f33e22-ce25-46bc-8add-b9aa4434e0e3-kube-api-access-7ks9s\") pod \"payload-processing-e2e-shared-b-w3-4f7628-7fd5f9b886-bgc5n\" (UID: \"a7f33e22-ce25-46bc-8add-b9aa4434e0e3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4f7628-7fd5f9b886-bgc5n" Oct 30 01:52:32.473606 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:32.473125 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7f33e22-ce25-46bc-8add-b9aa4434e0e3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-4f7628-7fd5f9b886-bgc5n\" (UID: \"a7f33e22-ce25-46bc-8add-b9aa4434e0e3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4f7628-7fd5f9b886-bgc5n" Oct 30 01:52:32.473606 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:32.473300 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbl4r\" (UniqueName: \"kubernetes.io/projected/7f16168b-61d0-4a08-94b7-7ea5581e4005-kube-api-access-nbl4r\") pod \"payload-pre-processing-e2e-shared-b-w3-4f7628-5d4595b5c5-q9f88\" (UID: \"7f16168b-61d0-4a08-94b7-7ea5581e4005\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4f7628-5d4595b5c5-q9f88" Oct 30 01:52:32.507555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:32.474809 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f16168b-61d0-4a08-94b7-7ea5581e4005-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-4f7628-5d4595b5c5-q9f88\" (UID: \"7f16168b-61d0-4a08-94b7-7ea5581e4005\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4f7628-5d4595b5c5-q9f88" Oct 30 01:52:32.507555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:32.475835 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7f33e22-ce25-46bc-8add-b9aa4434e0e3-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-4f7628-7fd5f9b886-bgc5n\" (UID: \"a7f33e22-ce25-46bc-8add-b9aa4434e0e3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4f7628-7fd5f9b886-bgc5n" Oct 30 01:52:32.548600 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:32.545296 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbl4r\" (UniqueName: \"kubernetes.io/projected/7f16168b-61d0-4a08-94b7-7ea5581e4005-kube-api-access-nbl4r\") pod \"payload-pre-processing-e2e-shared-b-w3-4f7628-5d4595b5c5-q9f88\" (UID: \"7f16168b-61d0-4a08-94b7-7ea5581e4005\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4f7628-5d4595b5c5-q9f88" Oct 30 01:52:32.567076 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:32.567014 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ks9s\" (UniqueName: \"kubernetes.io/projected/a7f33e22-ce25-46bc-8add-b9aa4434e0e3-kube-api-access-7ks9s\") pod \"payload-processing-e2e-shared-b-w3-4f7628-7fd5f9b886-bgc5n\" (UID: \"a7f33e22-ce25-46bc-8add-b9aa4434e0e3\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4f7628-7fd5f9b886-bgc5n" Oct 30 01:52:32.596889 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:32.596854 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4f7628-5d4595b5c5-q9f88" Oct 30 01:52:32.838075 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:32.838043 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4f7628-7fd5f9b886-bgc5n" Oct 30 01:52:32.915946 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:32.915913 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4f7628-5d4595b5c5-q9f88"] Oct 30 01:52:32.920800 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:52:32.920323 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f16168b_61d0_4a08_94b7_7ea5581e4005.slice/crio-f071e818f3b496b0ce6baf692330f16f71e76d2fa948a0f2628a73ae90ff5488 WatchSource:0}: Error finding container f071e818f3b496b0ce6baf692330f16f71e76d2fa948a0f2628a73ae90ff5488: Status 404 returned error can't find the container with id f071e818f3b496b0ce6baf692330f16f71e76d2fa948a0f2628a73ae90ff5488 Oct 30 01:52:33.181123 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:33.181085 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4f7628-7fd5f9b886-bgc5n"] Oct 30 01:52:33.677278 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:33.677244 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4f7628-7fd5f9b886-bgc5n" event={"ID":"a7f33e22-ce25-46bc-8add-b9aa4434e0e3","Type":"ContainerStarted","Data":"2039a470a99f4e5e0dd3385cc285ed722dbcca96a8fdd3576b5e8cf4e33f2416"} Oct 30 01:52:33.677811 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:33.677300 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4f7628-7fd5f9b886-bgc5n" event={"ID":"a7f33e22-ce25-46bc-8add-b9aa4434e0e3","Type":"ContainerStarted","Data":"ccf66000ba3884eda091d38f00f4bfcc1b96eccb662fec5230be5f04ee6147f2"} Oct 30 01:52:33.677811 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:33.677422 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4f7628-7fd5f9b886-bgc5n" Oct 30 01:52:33.678908 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:33.678887 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4f7628-5d4595b5c5-q9f88" event={"ID":"7f16168b-61d0-4a08-94b7-7ea5581e4005","Type":"ContainerStarted","Data":"c7f698cc3c62befb5e20a345d0255ba0e086dbff130aa1bf2f01bb13825a9328"} Oct 30 01:52:33.679032 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:33.678920 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4f7628-5d4595b5c5-q9f88" event={"ID":"7f16168b-61d0-4a08-94b7-7ea5581e4005","Type":"ContainerStarted","Data":"f071e818f3b496b0ce6baf692330f16f71e76d2fa948a0f2628a73ae90ff5488"} Oct 30 01:52:33.679032 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:33.678960 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4f7628-5d4595b5c5-q9f88" Oct 30 01:52:33.717860 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:33.717803 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4f7628-7fd5f9b886-bgc5n" podStartSLOduration=1.7177861380000001 podStartE2EDuration="1.717786138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:52:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:52:33.715747047 +0000 UTC m=+2077.294034384" watchObservedRunningTime="2026-10-30 01:52:33.717786138 +0000 UTC m=+2077.296073473" Oct 30 01:52:33.754604 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:33.754558 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4f7628-5d4595b5c5-q9f88" podStartSLOduration=1.754503675 podStartE2EDuration="1.754503675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:52:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:52:33.753861568 +0000 UTC m=+2077.332148904" watchObservedRunningTime="2026-10-30 01:52:33.754503675 +0000 UTC m=+2077.332791012" Oct 30 01:52:35.696925 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:35.696891 6221 generic.go:356] "Generic (PLEG): container finished" podID="40e8c5af-0173-4388-b938-ddc9b82ea112" containerID="ebef54a515eebaf98f3f0719c4d2c943c8b073037d929cad5d3f542928cc3c48" exitCode=0 Oct 30 01:52:35.697321 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:35.696967 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr" event={"ID":"40e8c5af-0173-4388-b938-ddc9b82ea112","Type":"ContainerDied","Data":"ebef54a515eebaf98f3f0719c4d2c943c8b073037d929cad5d3f542928cc3c48"} Oct 30 01:52:35.734104 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:35.734050 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr" Oct 30 01:52:35.828564 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:35.828502 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40e8c5af-0173-4388-b938-ddc9b82ea112-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e8c5af-0173-4388-b938-ddc9b82ea112-istio-data\") pod \"40e8c5af-0173-4388-b938-ddc9b82ea112\" (UID: \"40e8c5af-0173-4388-b938-ddc9b82ea112\") " Oct 30 01:52:35.828765 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:35.828585 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40e8c5af-0173-4388-b938-ddc9b82ea112-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e8c5af-0173-4388-b938-ddc9b82ea112-credential-socket\") pod \"40e8c5af-0173-4388-b938-ddc9b82ea112\" (UID: \"40e8c5af-0173-4388-b938-ddc9b82ea112\") " Oct 30 01:52:35.828765 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:35.828621 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40e8c5af-0173-4388-b938-ddc9b82ea112-istio-token\" (UniqueName: \"kubernetes.io/projected/40e8c5af-0173-4388-b938-ddc9b82ea112-istio-token\") pod \"40e8c5af-0173-4388-b938-ddc9b82ea112\" (UID: \"40e8c5af-0173-4388-b938-ddc9b82ea112\") " Oct 30 01:52:35.828765 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:35.828667 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40e8c5af-0173-4388-b938-ddc9b82ea112-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e8c5af-0173-4388-b938-ddc9b82ea112-workload-certs\") pod \"40e8c5af-0173-4388-b938-ddc9b82ea112\" (UID: \"40e8c5af-0173-4388-b938-ddc9b82ea112\") " Oct 30 01:52:35.828950 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:35.828780 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40e8c5af-0173-4388-b938-ddc9b82ea112-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e8c5af-0173-4388-b938-ddc9b82ea112-workload-socket\") pod \"40e8c5af-0173-4388-b938-ddc9b82ea112\" (UID: \"40e8c5af-0173-4388-b938-ddc9b82ea112\") " Oct 30 01:52:35.828950 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:35.828810 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e8c5af-0173-4388-b938-ddc9b82ea112-istio-data" pod "40e8c5af-0173-4388-b938-ddc9b82ea112" (UID: "40e8c5af-0173-4388-b938-ddc9b82ea112"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:35.828950 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:35.828831 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40e8c5af-0173-4388-b938-ddc9b82ea112-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e8c5af-0173-4388-b938-ddc9b82ea112-istio-podinfo\") pod \"40e8c5af-0173-4388-b938-ddc9b82ea112\" (UID: \"40e8c5af-0173-4388-b938-ddc9b82ea112\") " Oct 30 01:52:35.828950 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:35.828870 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40e8c5af-0173-4388-b938-ddc9b82ea112-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e8c5af-0173-4388-b938-ddc9b82ea112-istiod-ca-cert\") pod \"40e8c5af-0173-4388-b938-ddc9b82ea112\" (UID: \"40e8c5af-0173-4388-b938-ddc9b82ea112\") " Oct 30 01:52:35.828950 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:35.828896 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e8c5af-0173-4388-b938-ddc9b82ea112-credential-socket" pod "40e8c5af-0173-4388-b938-ddc9b82ea112" (UID: "40e8c5af-0173-4388-b938-ddc9b82ea112"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:35.829217 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:35.828976 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40e8c5af-0173-4388-b938-ddc9b82ea112-kube-api-access-jm97h\" (UniqueName: \"kubernetes.io/projected/40e8c5af-0173-4388-b938-ddc9b82ea112-kube-api-access-jm97h\") pod \"40e8c5af-0173-4388-b938-ddc9b82ea112\" (UID: \"40e8c5af-0173-4388-b938-ddc9b82ea112\") " Oct 30 01:52:35.829217 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:35.829018 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40e8c5af-0173-4388-b938-ddc9b82ea112-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e8c5af-0173-4388-b938-ddc9b82ea112-istio-envoy\") pod \"40e8c5af-0173-4388-b938-ddc9b82ea112\" (UID: \"40e8c5af-0173-4388-b938-ddc9b82ea112\") " Oct 30 01:52:35.829217 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:35.829071 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e8c5af-0173-4388-b938-ddc9b82ea112-workload-socket" pod "40e8c5af-0173-4388-b938-ddc9b82ea112" (UID: "40e8c5af-0173-4388-b938-ddc9b82ea112"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:35.829398 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:35.829371 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e8c5af-0173-4388-b938-ddc9b82ea112-workload-certs" pod "40e8c5af-0173-4388-b938-ddc9b82ea112" (UID: "40e8c5af-0173-4388-b938-ddc9b82ea112"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:35.829480 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:35.829391 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40e8c5af-0173-4388-b938-ddc9b82ea112-istiod-ca-cert" pod "40e8c5af-0173-4388-b938-ddc9b82ea112" (UID: "40e8c5af-0173-4388-b938-ddc9b82ea112"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:52:35.829768 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:35.829745 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e8c5af-0173-4388-b938-ddc9b82ea112-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:35.829941 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:35.829925 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40e8c5af-0173-4388-b938-ddc9b82ea112-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:35.830047 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:35.830036 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40e8c5af-0173-4388-b938-ddc9b82ea112-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:35.830156 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:35.830144 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40e8c5af-0173-4388-b938-ddc9b82ea112-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:35.830258 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:35.830244 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40e8c5af-0173-4388-b938-ddc9b82ea112-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:35.832198 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:35.831930 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40e8c5af-0173-4388-b938-ddc9b82ea112-istio-envoy" pod "40e8c5af-0173-4388-b938-ddc9b82ea112" (UID: "40e8c5af-0173-4388-b938-ddc9b82ea112"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:35.832198 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:35.831950 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40e8c5af-0173-4388-b938-ddc9b82ea112-istio-token" pod "40e8c5af-0173-4388-b938-ddc9b82ea112" (UID: "40e8c5af-0173-4388-b938-ddc9b82ea112"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:52:35.832198 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:35.832025 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40e8c5af-0173-4388-b938-ddc9b82ea112-istio-podinfo" pod "40e8c5af-0173-4388-b938-ddc9b82ea112" (UID: "40e8c5af-0173-4388-b938-ddc9b82ea112"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:52:35.832198 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:35.832144 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40e8c5af-0173-4388-b938-ddc9b82ea112-kube-api-access-jm97h" pod "40e8c5af-0173-4388-b938-ddc9b82ea112" (UID: "40e8c5af-0173-4388-b938-ddc9b82ea112"). InnerVolumeSpecName "kube-api-access-jm97h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:52:35.931331 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:35.931300 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40e8c5af-0173-4388-b938-ddc9b82ea112-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:35.931331 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:35.931324 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40e8c5af-0173-4388-b938-ddc9b82ea112-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:35.931331 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:35.931335 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm97h\" (UniqueName: \"kubernetes.io/projected/40e8c5af-0173-4388-b938-ddc9b82ea112-kube-api-access-jm97h\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:35.931602 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:35.931348 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40e8c5af-0173-4388-b938-ddc9b82ea112-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:36.705769 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:36.705732 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr" event={"ID":"40e8c5af-0173-4388-b938-ddc9b82ea112","Type":"ContainerDied","Data":"c0f4bd7191be185a2dc6ec43f6f5821ed920388c518618c942ed3076eb95abca"} Oct 30 01:52:36.706196 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:36.705797 6221 scope.go:122] "RemoveContainer" containerID="ebef54a515eebaf98f3f0719c4d2c943c8b073037d929cad5d3f542928cc3c48" Oct 30 01:52:36.706196 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:36.705745 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr" Oct 30 01:52:36.723213 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:36.723177 6221 patch_prober.go:29] interesting pod/e2e-worker-w3-558327-openshift-default-576847689f-flsxr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 01:52:36.723324 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:36.723231 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr" podUID="40e8c5af-0173-4388-b938-ddc9b82ea112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 01:52:36.772385 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:36.772356 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr"] Oct 30 01:52:36.784695 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:36.784667 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-flsxr"] Oct 30 01:52:38.547558 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:38.547450 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40e8c5af-0173-4388-b938-ddc9b82ea112" path="/var/lib/kubelet/pods/40e8c5af-0173-4388-b938-ddc9b82ea112/volumes" Oct 30 01:52:44.690169 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:44.690136 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4f7628-5d4595b5c5-q9f88" Oct 30 01:52:44.690717 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:44.690228 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4f7628-7fd5f9b886-bgc5n" Oct 30 01:52:48.487561 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:48.487514 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-861cd2/auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5"] Oct 30 01:52:50.521929 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:50.521894 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x"] Oct 30 01:52:50.522408 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:50.522092 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x" podUID="e0f654ff-538a-41ae-ab41-f3e8d658949d" containerName="istio-proxy" containerID="cri-o://86f18a15f86d84107977e6620573b962bc905c28fbcbbd5dd1c50818c2a8d6b9" gracePeriod=30 Oct 30 01:52:50.534318 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:50.534291 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz"] Oct 30 01:52:50.534568 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:50.534517 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz" podUID="12605667-e208-4a48-bb1b-61d1e76066bf" containerName="istio-proxy" containerID="cri-o://694a9016590c12c573bc946f60effc9d71f0397f73b59ab6588b47fba1fe6ed2" gracePeriod=30 Oct 30 01:52:50.537865 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:50.537839 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f"] Oct 30 01:52:50.538067 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:50.538028 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f" podUID="f347ecd6-e2d7-43da-ba3e-88006f24c167" containerName="istio-proxy" containerID="cri-o://091691472dff64c611649bc7c4dfcbd1d44e80ec09d0e24a27c69311e45297f6" gracePeriod=30 Oct 30 01:52:51.494730 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:52:51.494695 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:52:55.006112 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:55.006071 6221 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 01:52:55.006575 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:55.006148 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz" podUID="12605667-e208-4a48-bb1b-61d1e76066bf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 01:52:55.849248 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:55.849214 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f" Oct 30 01:52:55.874183 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:55.874150 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz" Oct 30 01:52:55.904715 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:55.904690 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x" Oct 30 01:52:55.935873 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:55.935839 6221 generic.go:356] "Generic (PLEG): container finished" podID="f347ecd6-e2d7-43da-ba3e-88006f24c167" containerID="091691472dff64c611649bc7c4dfcbd1d44e80ec09d0e24a27c69311e45297f6" exitCode=0 Oct 30 01:52:55.936072 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:55.935882 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f" Oct 30 01:52:55.936072 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:55.935918 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f" event={"ID":"f347ecd6-e2d7-43da-ba3e-88006f24c167","Type":"ContainerDied","Data":"091691472dff64c611649bc7c4dfcbd1d44e80ec09d0e24a27c69311e45297f6"} Oct 30 01:52:55.936072 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:55.935969 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f" event={"ID":"f347ecd6-e2d7-43da-ba3e-88006f24c167","Type":"ContainerDied","Data":"29527bd4e551b2d2dd86e3d7767cbb0c792f595fb4c5ff45e150f6b7ae87793d"} Oct 30 01:52:55.936072 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:55.935999 6221 scope.go:122] "RemoveContainer" containerID="091691472dff64c611649bc7c4dfcbd1d44e80ec09d0e24a27c69311e45297f6" Oct 30 01:52:55.937631 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:55.937610 6221 generic.go:356] "Generic (PLEG): container finished" podID="12605667-e208-4a48-bb1b-61d1e76066bf" containerID="694a9016590c12c573bc946f60effc9d71f0397f73b59ab6588b47fba1fe6ed2" exitCode=0 Oct 30 01:52:55.937743 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:55.937654 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz" Oct 30 01:52:55.937743 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:55.937695 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz" event={"ID":"12605667-e208-4a48-bb1b-61d1e76066bf","Type":"ContainerDied","Data":"694a9016590c12c573bc946f60effc9d71f0397f73b59ab6588b47fba1fe6ed2"} Oct 30 01:52:55.937871 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:55.937741 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz" event={"ID":"12605667-e208-4a48-bb1b-61d1e76066bf","Type":"ContainerDied","Data":"60d64cd87f3ab5f263f1b6e8819ced495fbe40b6338ea7baaeb3ddb744550e9d"} Oct 30 01:52:55.939281 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:55.939256 6221 generic.go:356] "Generic (PLEG): container finished" podID="e0f654ff-538a-41ae-ab41-f3e8d658949d" containerID="86f18a15f86d84107977e6620573b962bc905c28fbcbbd5dd1c50818c2a8d6b9" exitCode=0 Oct 30 01:52:55.939414 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:55.939350 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x" event={"ID":"e0f654ff-538a-41ae-ab41-f3e8d658949d","Type":"ContainerDied","Data":"86f18a15f86d84107977e6620573b962bc905c28fbcbbd5dd1c50818c2a8d6b9"} Oct 30 01:52:55.939414 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:55.939389 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x" event={"ID":"e0f654ff-538a-41ae-ab41-f3e8d658949d","Type":"ContainerDied","Data":"42a2e1034d153cf2da116c9d64f3adad2ef1605f66f4b1840abaf7057b9fa5dc"} Oct 30 01:52:55.939658 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:55.939633 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x" Oct 30 01:52:55.948916 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:55.948890 6221 scope.go:122] "RemoveContainer" containerID="091691472dff64c611649bc7c4dfcbd1d44e80ec09d0e24a27c69311e45297f6" Oct 30 01:52:55.949281 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:52:55.949261 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"091691472dff64c611649bc7c4dfcbd1d44e80ec09d0e24a27c69311e45297f6\": container with ID starting with 091691472dff64c611649bc7c4dfcbd1d44e80ec09d0e24a27c69311e45297f6 not found: ID does not exist" containerID="091691472dff64c611649bc7c4dfcbd1d44e80ec09d0e24a27c69311e45297f6" Oct 30 01:52:55.949373 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:55.949292 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"091691472dff64c611649bc7c4dfcbd1d44e80ec09d0e24a27c69311e45297f6"} err="failed to get container status \"091691472dff64c611649bc7c4dfcbd1d44e80ec09d0e24a27c69311e45297f6\": rpc error: code = NotFound desc = could not find container \"091691472dff64c611649bc7c4dfcbd1d44e80ec09d0e24a27c69311e45297f6\": container with ID starting with 091691472dff64c611649bc7c4dfcbd1d44e80ec09d0e24a27c69311e45297f6 not found: ID does not exist" Oct 30 01:52:55.949373 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:55.949319 6221 scope.go:122] "RemoveContainer" containerID="694a9016590c12c573bc946f60effc9d71f0397f73b59ab6588b47fba1fe6ed2" Oct 30 01:52:55.970338 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:55.970315 6221 scope.go:122] "RemoveContainer" containerID="694a9016590c12c573bc946f60effc9d71f0397f73b59ab6588b47fba1fe6ed2" Oct 30 01:52:55.970717 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:52:55.970692 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"694a9016590c12c573bc946f60effc9d71f0397f73b59ab6588b47fba1fe6ed2\": container with ID starting with 694a9016590c12c573bc946f60effc9d71f0397f73b59ab6588b47fba1fe6ed2 not found: ID does not exist" containerID="694a9016590c12c573bc946f60effc9d71f0397f73b59ab6588b47fba1fe6ed2" Oct 30 01:52:55.970807 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:55.970725 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"694a9016590c12c573bc946f60effc9d71f0397f73b59ab6588b47fba1fe6ed2"} err="failed to get container status \"694a9016590c12c573bc946f60effc9d71f0397f73b59ab6588b47fba1fe6ed2\": rpc error: code = NotFound desc = could not find container \"694a9016590c12c573bc946f60effc9d71f0397f73b59ab6588b47fba1fe6ed2\": container with ID starting with 694a9016590c12c573bc946f60effc9d71f0397f73b59ab6588b47fba1fe6ed2 not found: ID does not exist" Oct 30 01:52:55.970807 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:55.970742 6221 scope.go:122] "RemoveContainer" containerID="86f18a15f86d84107977e6620573b962bc905c28fbcbbd5dd1c50818c2a8d6b9" Oct 30 01:52:55.981607 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:55.981570 6221 scope.go:122] "RemoveContainer" containerID="86f18a15f86d84107977e6620573b962bc905c28fbcbbd5dd1c50818c2a8d6b9" Oct 30 01:52:55.981961 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:52:55.981935 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86f18a15f86d84107977e6620573b962bc905c28fbcbbd5dd1c50818c2a8d6b9\": container with ID starting with 86f18a15f86d84107977e6620573b962bc905c28fbcbbd5dd1c50818c2a8d6b9 not found: ID does not exist" containerID="86f18a15f86d84107977e6620573b962bc905c28fbcbbd5dd1c50818c2a8d6b9" Oct 30 01:52:55.982054 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:55.981969 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86f18a15f86d84107977e6620573b962bc905c28fbcbbd5dd1c50818c2a8d6b9"} err="failed to get container status \"86f18a15f86d84107977e6620573b962bc905c28fbcbbd5dd1c50818c2a8d6b9\": rpc error: code = NotFound desc = could not find container \"86f18a15f86d84107977e6620573b962bc905c28fbcbbd5dd1c50818c2a8d6b9\": container with ID starting with 86f18a15f86d84107977e6620573b962bc905c28fbcbbd5dd1c50818c2a8d6b9 not found: ID does not exist" Oct 30 01:52:56.014820 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.014789 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f347ecd6-e2d7-43da-ba3e-88006f24c167-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f347ecd6-e2d7-43da-ba3e-88006f24c167-credential-socket\") pod \"f347ecd6-e2d7-43da-ba3e-88006f24c167\" (UID: \"f347ecd6-e2d7-43da-ba3e-88006f24c167\") " Oct 30 01:52:56.015275 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.014830 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0f654ff-538a-41ae-ab41-f3e8d658949d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0f654ff-538a-41ae-ab41-f3e8d658949d-istio-envoy\") pod \"e0f654ff-538a-41ae-ab41-f3e8d658949d\" (UID: \"e0f654ff-538a-41ae-ab41-f3e8d658949d\") " Oct 30 01:52:56.015275 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.014865 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0f654ff-538a-41ae-ab41-f3e8d658949d-istio-token\" (UniqueName: \"kubernetes.io/projected/e0f654ff-538a-41ae-ab41-f3e8d658949d-istio-token\") pod \"e0f654ff-538a-41ae-ab41-f3e8d658949d\" (UID: \"e0f654ff-538a-41ae-ab41-f3e8d658949d\") " Oct 30 01:52:56.015275 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.014903 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0f654ff-538a-41ae-ab41-f3e8d658949d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0f654ff-538a-41ae-ab41-f3e8d658949d-workload-certs\") pod \"e0f654ff-538a-41ae-ab41-f3e8d658949d\" (UID: \"e0f654ff-538a-41ae-ab41-f3e8d658949d\") " Oct 30 01:52:56.015275 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.014959 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0f654ff-538a-41ae-ab41-f3e8d658949d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0f654ff-538a-41ae-ab41-f3e8d658949d-credential-socket\") pod \"e0f654ff-538a-41ae-ab41-f3e8d658949d\" (UID: \"e0f654ff-538a-41ae-ab41-f3e8d658949d\") " Oct 30 01:52:56.015275 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.014993 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0f654ff-538a-41ae-ab41-f3e8d658949d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0f654ff-538a-41ae-ab41-f3e8d658949d-workload-socket\") pod \"e0f654ff-538a-41ae-ab41-f3e8d658949d\" (UID: \"e0f654ff-538a-41ae-ab41-f3e8d658949d\") " Oct 30 01:52:56.015275 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.015071 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12605667-e208-4a48-bb1b-61d1e76066bf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12605667-e208-4a48-bb1b-61d1e76066bf-credential-socket\") pod \"12605667-e208-4a48-bb1b-61d1e76066bf\" (UID: \"12605667-e208-4a48-bb1b-61d1e76066bf\") " Oct 30 01:52:56.015275 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.015119 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0f654ff-538a-41ae-ab41-f3e8d658949d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0f654ff-538a-41ae-ab41-f3e8d658949d-istiod-ca-cert\") pod \"e0f654ff-538a-41ae-ab41-f3e8d658949d\" (UID: \"e0f654ff-538a-41ae-ab41-f3e8d658949d\") " Oct 30 01:52:56.015275 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.015138 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f347ecd6-e2d7-43da-ba3e-88006f24c167-credential-socket" pod "f347ecd6-e2d7-43da-ba3e-88006f24c167" (UID: "f347ecd6-e2d7-43da-ba3e-88006f24c167"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:56.015275 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.015148 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f347ecd6-e2d7-43da-ba3e-88006f24c167-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f347ecd6-e2d7-43da-ba3e-88006f24c167-workload-certs\") pod \"f347ecd6-e2d7-43da-ba3e-88006f24c167\" (UID: \"f347ecd6-e2d7-43da-ba3e-88006f24c167\") " Oct 30 01:52:56.015275 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.015197 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0f654ff-538a-41ae-ab41-f3e8d658949d-workload-certs" pod "e0f654ff-538a-41ae-ab41-f3e8d658949d" (UID: "e0f654ff-538a-41ae-ab41-f3e8d658949d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:56.015275 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.015243 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f347ecd6-e2d7-43da-ba3e-88006f24c167-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f347ecd6-e2d7-43da-ba3e-88006f24c167-istio-podinfo\") pod \"f347ecd6-e2d7-43da-ba3e-88006f24c167\" (UID: \"f347ecd6-e2d7-43da-ba3e-88006f24c167\") " Oct 30 01:52:56.015935 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.015320 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12605667-e208-4a48-bb1b-61d1e76066bf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12605667-e208-4a48-bb1b-61d1e76066bf-workload-certs\") pod \"12605667-e208-4a48-bb1b-61d1e76066bf\" (UID: \"12605667-e208-4a48-bb1b-61d1e76066bf\") " Oct 30 01:52:56.015935 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.015367 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0f654ff-538a-41ae-ab41-f3e8d658949d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0f654ff-538a-41ae-ab41-f3e8d658949d-istio-podinfo\") pod \"e0f654ff-538a-41ae-ab41-f3e8d658949d\" (UID: \"e0f654ff-538a-41ae-ab41-f3e8d658949d\") " Oct 30 01:52:56.015935 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.015368 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f347ecd6-e2d7-43da-ba3e-88006f24c167-workload-certs" pod "f347ecd6-e2d7-43da-ba3e-88006f24c167" (UID: "f347ecd6-e2d7-43da-ba3e-88006f24c167"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:56.015935 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.015396 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0f654ff-538a-41ae-ab41-f3e8d658949d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0f654ff-538a-41ae-ab41-f3e8d658949d-istio-data\") pod \"e0f654ff-538a-41ae-ab41-f3e8d658949d\" (UID: \"e0f654ff-538a-41ae-ab41-f3e8d658949d\") " Oct 30 01:52:56.015935 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.015443 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12605667-e208-4a48-bb1b-61d1e76066bf-credential-socket" pod "12605667-e208-4a48-bb1b-61d1e76066bf" (UID: "12605667-e208-4a48-bb1b-61d1e76066bf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:56.015935 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.015447 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12605667-e208-4a48-bb1b-61d1e76066bf-istio-token\" (UniqueName: \"kubernetes.io/projected/12605667-e208-4a48-bb1b-61d1e76066bf-istio-token\") pod \"12605667-e208-4a48-bb1b-61d1e76066bf\" (UID: \"12605667-e208-4a48-bb1b-61d1e76066bf\") " Oct 30 01:52:56.015935 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.015497 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12605667-e208-4a48-bb1b-61d1e76066bf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12605667-e208-4a48-bb1b-61d1e76066bf-istiod-ca-cert\") pod \"12605667-e208-4a48-bb1b-61d1e76066bf\" (UID: \"12605667-e208-4a48-bb1b-61d1e76066bf\") " Oct 30 01:52:56.015935 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.015589 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12605667-e208-4a48-bb1b-61d1e76066bf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12605667-e208-4a48-bb1b-61d1e76066bf-workload-socket\") pod \"12605667-e208-4a48-bb1b-61d1e76066bf\" (UID: \"12605667-e208-4a48-bb1b-61d1e76066bf\") " Oct 30 01:52:56.015935 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.015596 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0f654ff-538a-41ae-ab41-f3e8d658949d-workload-socket" pod "e0f654ff-538a-41ae-ab41-f3e8d658949d" (UID: "e0f654ff-538a-41ae-ab41-f3e8d658949d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:56.015935 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.015606 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12605667-e208-4a48-bb1b-61d1e76066bf-workload-certs" pod "12605667-e208-4a48-bb1b-61d1e76066bf" (UID: "12605667-e208-4a48-bb1b-61d1e76066bf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:56.015935 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.015630 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12605667-e208-4a48-bb1b-61d1e76066bf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12605667-e208-4a48-bb1b-61d1e76066bf-istio-envoy\") pod \"12605667-e208-4a48-bb1b-61d1e76066bf\" (UID: \"12605667-e208-4a48-bb1b-61d1e76066bf\") " Oct 30 01:52:56.015935 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.015673 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f347ecd6-e2d7-43da-ba3e-88006f24c167-istio-token\" (UniqueName: \"kubernetes.io/projected/f347ecd6-e2d7-43da-ba3e-88006f24c167-istio-token\") pod \"f347ecd6-e2d7-43da-ba3e-88006f24c167\" (UID: \"f347ecd6-e2d7-43da-ba3e-88006f24c167\") " Oct 30 01:52:56.015935 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.015707 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f347ecd6-e2d7-43da-ba3e-88006f24c167-kube-api-access-jmprs\" (UniqueName: \"kubernetes.io/projected/f347ecd6-e2d7-43da-ba3e-88006f24c167-kube-api-access-jmprs\") pod \"f347ecd6-e2d7-43da-ba3e-88006f24c167\" (UID: \"f347ecd6-e2d7-43da-ba3e-88006f24c167\") " Oct 30 01:52:56.015935 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.015742 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f347ecd6-e2d7-43da-ba3e-88006f24c167-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f347ecd6-e2d7-43da-ba3e-88006f24c167-istio-envoy\") pod \"f347ecd6-e2d7-43da-ba3e-88006f24c167\" (UID: \"f347ecd6-e2d7-43da-ba3e-88006f24c167\") " Oct 30 01:52:56.015935 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.015810 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0f654ff-538a-41ae-ab41-f3e8d658949d-kube-api-access-m9vtq\" (UniqueName: \"kubernetes.io/projected/e0f654ff-538a-41ae-ab41-f3e8d658949d-kube-api-access-m9vtq\") pod \"e0f654ff-538a-41ae-ab41-f3e8d658949d\" (UID: \"e0f654ff-538a-41ae-ab41-f3e8d658949d\") " Oct 30 01:52:56.015935 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.015839 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f347ecd6-e2d7-43da-ba3e-88006f24c167-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f347ecd6-e2d7-43da-ba3e-88006f24c167-istio-data\") pod \"f347ecd6-e2d7-43da-ba3e-88006f24c167\" (UID: \"f347ecd6-e2d7-43da-ba3e-88006f24c167\") " Oct 30 01:52:56.015935 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.015886 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12605667-e208-4a48-bb1b-61d1e76066bf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12605667-e208-4a48-bb1b-61d1e76066bf-istio-podinfo\") pod \"12605667-e208-4a48-bb1b-61d1e76066bf\" (UID: \"12605667-e208-4a48-bb1b-61d1e76066bf\") " Oct 30 01:52:56.015935 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.015912 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0f654ff-538a-41ae-ab41-f3e8d658949d-istiod-ca-cert" pod "e0f654ff-538a-41ae-ab41-f3e8d658949d" (UID: "e0f654ff-538a-41ae-ab41-f3e8d658949d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:52:56.015935 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.015923 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12605667-e208-4a48-bb1b-61d1e76066bf-kube-api-access-7jpnm\" (UniqueName: \"kubernetes.io/projected/12605667-e208-4a48-bb1b-61d1e76066bf-kube-api-access-7jpnm\") pod \"12605667-e208-4a48-bb1b-61d1e76066bf\" (UID: \"12605667-e208-4a48-bb1b-61d1e76066bf\") " Oct 30 01:52:56.016955 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.015968 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f347ecd6-e2d7-43da-ba3e-88006f24c167-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f347ecd6-e2d7-43da-ba3e-88006f24c167-istiod-ca-cert\") pod \"f347ecd6-e2d7-43da-ba3e-88006f24c167\" (UID: \"f347ecd6-e2d7-43da-ba3e-88006f24c167\") " Oct 30 01:52:56.016955 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.016103 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f347ecd6-e2d7-43da-ba3e-88006f24c167-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f347ecd6-e2d7-43da-ba3e-88006f24c167-workload-socket\") pod \"f347ecd6-e2d7-43da-ba3e-88006f24c167\" (UID: \"f347ecd6-e2d7-43da-ba3e-88006f24c167\") " Oct 30 01:52:56.016955 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.016137 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0f654ff-538a-41ae-ab41-f3e8d658949d-credential-socket" pod "e0f654ff-538a-41ae-ab41-f3e8d658949d" (UID: "e0f654ff-538a-41ae-ab41-f3e8d658949d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:56.016955 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.016150 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12605667-e208-4a48-bb1b-61d1e76066bf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12605667-e208-4a48-bb1b-61d1e76066bf-istio-data\") pod \"12605667-e208-4a48-bb1b-61d1e76066bf\" (UID: \"12605667-e208-4a48-bb1b-61d1e76066bf\") " Oct 30 01:52:56.016955 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.016517 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12605667-e208-4a48-bb1b-61d1e76066bf-istiod-ca-cert" pod "12605667-e208-4a48-bb1b-61d1e76066bf" (UID: "12605667-e208-4a48-bb1b-61d1e76066bf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:52:56.016955 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.016744 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12605667-e208-4a48-bb1b-61d1e76066bf-istio-data" pod "12605667-e208-4a48-bb1b-61d1e76066bf" (UID: "12605667-e208-4a48-bb1b-61d1e76066bf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:56.017294 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.017176 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0f654ff-538a-41ae-ab41-f3e8d658949d-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:56.017294 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.017207 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0f654ff-538a-41ae-ab41-f3e8d658949d-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:56.017294 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.017222 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12605667-e208-4a48-bb1b-61d1e76066bf-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:56.017294 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.017236 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0f654ff-538a-41ae-ab41-f3e8d658949d-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:56.017294 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.017251 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f347ecd6-e2d7-43da-ba3e-88006f24c167-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:56.017294 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.017279 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12605667-e208-4a48-bb1b-61d1e76066bf-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:56.017294 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.017291 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12605667-e208-4a48-bb1b-61d1e76066bf-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:56.017779 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.017303 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12605667-e208-4a48-bb1b-61d1e76066bf-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:56.017779 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.017318 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f347ecd6-e2d7-43da-ba3e-88006f24c167-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:56.017779 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.017332 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0f654ff-538a-41ae-ab41-f3e8d658949d-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:56.017973 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.017952 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0f654ff-538a-41ae-ab41-f3e8d658949d-istio-token" pod "e0f654ff-538a-41ae-ab41-f3e8d658949d" (UID: "e0f654ff-538a-41ae-ab41-f3e8d658949d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:52:56.018108 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.018081 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0f654ff-538a-41ae-ab41-f3e8d658949d-istio-data" pod "e0f654ff-538a-41ae-ab41-f3e8d658949d" (UID: "e0f654ff-538a-41ae-ab41-f3e8d658949d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:56.018236 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.018146 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12605667-e208-4a48-bb1b-61d1e76066bf-workload-socket" pod "12605667-e208-4a48-bb1b-61d1e76066bf" (UID: "12605667-e208-4a48-bb1b-61d1e76066bf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:56.018315 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.018244 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f347ecd6-e2d7-43da-ba3e-88006f24c167-istiod-ca-cert" pod "f347ecd6-e2d7-43da-ba3e-88006f24c167" (UID: "f347ecd6-e2d7-43da-ba3e-88006f24c167"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:52:56.018378 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.018330 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f347ecd6-e2d7-43da-ba3e-88006f24c167-workload-socket" pod "f347ecd6-e2d7-43da-ba3e-88006f24c167" (UID: "f347ecd6-e2d7-43da-ba3e-88006f24c167"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:56.019373 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.019095 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f347ecd6-e2d7-43da-ba3e-88006f24c167-kube-api-access-jmprs" pod "f347ecd6-e2d7-43da-ba3e-88006f24c167" (UID: "f347ecd6-e2d7-43da-ba3e-88006f24c167"). InnerVolumeSpecName "kube-api-access-jmprs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:52:56.019373 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.019184 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f347ecd6-e2d7-43da-ba3e-88006f24c167-istio-data" pod "f347ecd6-e2d7-43da-ba3e-88006f24c167" (UID: "f347ecd6-e2d7-43da-ba3e-88006f24c167"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:56.019598 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.019578 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12605667-e208-4a48-bb1b-61d1e76066bf-istio-token" pod "12605667-e208-4a48-bb1b-61d1e76066bf" (UID: "12605667-e208-4a48-bb1b-61d1e76066bf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:52:56.030564 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.019743 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f347ecd6-e2d7-43da-ba3e-88006f24c167-istio-podinfo" pod "f347ecd6-e2d7-43da-ba3e-88006f24c167" (UID: "f347ecd6-e2d7-43da-ba3e-88006f24c167"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:52:56.030564 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.020137 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0f654ff-538a-41ae-ab41-f3e8d658949d-istio-podinfo" pod "e0f654ff-538a-41ae-ab41-f3e8d658949d" (UID: "e0f654ff-538a-41ae-ab41-f3e8d658949d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:52:56.030564 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.021272 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0f654ff-538a-41ae-ab41-f3e8d658949d-kube-api-access-m9vtq" pod "e0f654ff-538a-41ae-ab41-f3e8d658949d" (UID: "e0f654ff-538a-41ae-ab41-f3e8d658949d"). InnerVolumeSpecName "kube-api-access-m9vtq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:52:56.030564 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.022252 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12605667-e208-4a48-bb1b-61d1e76066bf-kube-api-access-7jpnm" pod "12605667-e208-4a48-bb1b-61d1e76066bf" (UID: "12605667-e208-4a48-bb1b-61d1e76066bf"). InnerVolumeSpecName "kube-api-access-7jpnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:52:56.030564 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.023759 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f347ecd6-e2d7-43da-ba3e-88006f24c167-istio-token" pod "f347ecd6-e2d7-43da-ba3e-88006f24c167" (UID: "f347ecd6-e2d7-43da-ba3e-88006f24c167"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:52:56.030564 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.023762 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12605667-e208-4a48-bb1b-61d1e76066bf-istio-podinfo" pod "12605667-e208-4a48-bb1b-61d1e76066bf" (UID: "12605667-e208-4a48-bb1b-61d1e76066bf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:52:56.030564 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.025180 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0f654ff-538a-41ae-ab41-f3e8d658949d-istio-envoy" pod "e0f654ff-538a-41ae-ab41-f3e8d658949d" (UID: "e0f654ff-538a-41ae-ab41-f3e8d658949d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:56.030564 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.026735 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12605667-e208-4a48-bb1b-61d1e76066bf-istio-envoy" pod "12605667-e208-4a48-bb1b-61d1e76066bf" (UID: "12605667-e208-4a48-bb1b-61d1e76066bf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:56.032935 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.032903 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f347ecd6-e2d7-43da-ba3e-88006f24c167-istio-envoy" pod "f347ecd6-e2d7-43da-ba3e-88006f24c167" (UID: "f347ecd6-e2d7-43da-ba3e-88006f24c167"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:52:56.124979 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.124900 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0f654ff-538a-41ae-ab41-f3e8d658949d-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:56.124979 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.124930 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0f654ff-538a-41ae-ab41-f3e8d658949d-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:56.124979 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.124939 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f347ecd6-e2d7-43da-ba3e-88006f24c167-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:56.124979 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.124947 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0f654ff-538a-41ae-ab41-f3e8d658949d-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:56.124979 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.124955 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0f654ff-538a-41ae-ab41-f3e8d658949d-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:56.124979 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.124963 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12605667-e208-4a48-bb1b-61d1e76066bf-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:56.124979 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.124971 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12605667-e208-4a48-bb1b-61d1e76066bf-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:56.124979 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.124978 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12605667-e208-4a48-bb1b-61d1e76066bf-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:56.124979 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.124986 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f347ecd6-e2d7-43da-ba3e-88006f24c167-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:56.125512 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.124994 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmprs\" (UniqueName: \"kubernetes.io/projected/f347ecd6-e2d7-43da-ba3e-88006f24c167-kube-api-access-jmprs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:56.125512 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.125004 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f347ecd6-e2d7-43da-ba3e-88006f24c167-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:56.125512 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.125013 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9vtq\" (UniqueName: \"kubernetes.io/projected/e0f654ff-538a-41ae-ab41-f3e8d658949d-kube-api-access-m9vtq\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:56.125512 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.125020 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f347ecd6-e2d7-43da-ba3e-88006f24c167-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:56.125512 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.125029 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12605667-e208-4a48-bb1b-61d1e76066bf-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:56.125512 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.125036 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jpnm\" (UniqueName: \"kubernetes.io/projected/12605667-e208-4a48-bb1b-61d1e76066bf-kube-api-access-7jpnm\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:56.125512 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.125051 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f347ecd6-e2d7-43da-ba3e-88006f24c167-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:56.125512 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.125063 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f347ecd6-e2d7-43da-ba3e-88006f24c167-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:52:56.290431 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.290394 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f"] Oct 30 01:52:56.294071 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.294041 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-c9g7f"] Oct 30 01:52:56.335921 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.335887 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz"] Oct 30 01:52:56.345836 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.345797 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-q2zgz"] Oct 30 01:52:56.370596 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.370560 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f7628-5cc968cb6c-644x7"] Oct 30 01:52:56.371439 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.371423 6221 cpu_manager.go:414] "containerMap: removing container" podUID="e0f654ff-538a-41ae-ab41-f3e8d658949d" containerName="istio-proxy" Oct 30 01:52:56.371496 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.371442 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0f654ff-538a-41ae-ab41-f3e8d658949d" containerName="istio-proxy" Oct 30 01:52:56.371496 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.371467 6221 cpu_manager.go:414] "containerMap: removing container" podUID="40e8c5af-0173-4388-b938-ddc9b82ea112" containerName="istio-proxy" Oct 30 01:52:56.371496 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.371474 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40e8c5af-0173-4388-b938-ddc9b82ea112" containerName="istio-proxy" Oct 30 01:52:56.371496 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.371489 6221 cpu_manager.go:414] "containerMap: removing container" podUID="f347ecd6-e2d7-43da-ba3e-88006f24c167" containerName="istio-proxy" Oct 30 01:52:56.371649 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.371497 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f347ecd6-e2d7-43da-ba3e-88006f24c167" containerName="istio-proxy" Oct 30 01:52:56.371649 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.371544 6221 cpu_manager.go:414] "containerMap: removing container" podUID="12605667-e208-4a48-bb1b-61d1e76066bf" containerName="istio-proxy" Oct 30 01:52:56.371649 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.371553 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12605667-e208-4a48-bb1b-61d1e76066bf" containerName="istio-proxy" Oct 30 01:52:56.371742 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.371732 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="12605667-e208-4a48-bb1b-61d1e76066bf" containerName="istio-proxy" Oct 30 01:52:56.371776 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.371753 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0f654ff-538a-41ae-ab41-f3e8d658949d" containerName="istio-proxy" Oct 30 01:52:56.371776 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.371764 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="40e8c5af-0173-4388-b938-ddc9b82ea112" containerName="istio-proxy" Oct 30 01:52:56.371837 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.371774 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="f347ecd6-e2d7-43da-ba3e-88006f24c167" containerName="istio-proxy" Oct 30 01:52:56.378624 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.378321 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f7628-5cc968cb6c-644x7" Oct 30 01:52:56.387484 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.387456 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x"] Oct 30 01:52:56.390790 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.390759 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-fsb4x"] Oct 30 01:52:56.396013 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.395982 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f7628-5cc968cb6c-644x7"] Oct 30 01:52:56.397141 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.397120 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-4f7628-serving-cert\"" Oct 30 01:52:56.448785 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.448748 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-92f0cc-7865694dd6-98f7j"] Oct 30 01:52:56.453240 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.453213 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-92f0cc-7865694dd6-98f7j" Oct 30 01:52:56.469402 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.469377 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-92f0cc\"" Oct 30 01:52:56.470011 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.469993 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-92f0cc-7865694dd6-98f7j"] Oct 30 01:52:56.493033 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.492993 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-92f0cc-74f74b5d79-whvzj"] Oct 30 01:52:56.497976 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.497949 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-92f0cc-74f74b5d79-whvzj" Oct 30 01:52:56.516686 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.515826 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-92f0cc-dockercfg-z6n2x\"" Oct 30 01:52:56.517141 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.517119 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-92f0cc-74f74b5d79-whvzj"] Oct 30 01:52:56.530499 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.530460 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-4f7628-5cc968cb6c-644x7\" (UID: \"d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f7628-5cc968cb6c-644x7" Oct 30 01:52:56.530693 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.530620 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf488\" (UniqueName: \"kubernetes.io/projected/d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2-kube-api-access-rf488\") pod \"maas-api-e2e-shared-b-w3-4f7628-5cc968cb6c-644x7\" (UID: \"d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f7628-5cc968cb6c-644x7" Oct 30 01:52:56.530799 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.530781 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-4f7628-5cc968cb6c-644x7\" (UID: \"d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f7628-5cc968cb6c-644x7" Oct 30 01:52:56.530911 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.530895 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-4f7628-5cc968cb6c-644x7\" (UID: \"d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f7628-5cc968cb6c-644x7" Oct 30 01:52:56.550976 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.550938 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12605667-e208-4a48-bb1b-61d1e76066bf" path="/var/lib/kubelet/pods/12605667-e208-4a48-bb1b-61d1e76066bf/volumes" Oct 30 01:52:56.551481 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.551460 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0f654ff-538a-41ae-ab41-f3e8d658949d" path="/var/lib/kubelet/pods/e0f654ff-538a-41ae-ab41-f3e8d658949d/volumes" Oct 30 01:52:56.552000 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.551981 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f347ecd6-e2d7-43da-ba3e-88006f24c167" path="/var/lib/kubelet/pods/f347ecd6-e2d7-43da-ba3e-88006f24c167/volumes" Oct 30 01:52:56.650603 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.648197 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-4f7628-5cc968cb6c-644x7\" (UID: \"d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f7628-5cc968cb6c-644x7" Oct 30 01:52:56.650603 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.648279 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68jl9\" (UniqueName: \"kubernetes.io/projected/f5083acb-200e-4c21-8879-b7a1ebc8d337-kube-api-access-68jl9\") pod \"payload-processing-e2e-extmod-a-92f0cc-7865694dd6-98f7j\" (UID: \"f5083acb-200e-4c21-8879-b7a1ebc8d337\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-92f0cc-7865694dd6-98f7j" Oct 30 01:52:56.650603 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.648341 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89nwx\" (UniqueName: \"kubernetes.io/projected/df8ed3ef-12e5-40a7-9322-4b2a5f921d60-kube-api-access-89nwx\") pod \"payload-pre-processing-e2e-extmod-a-92f0cc-74f74b5d79-whvzj\" (UID: \"df8ed3ef-12e5-40a7-9322-4b2a5f921d60\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-92f0cc-74f74b5d79-whvzj" Oct 30 01:52:56.650603 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.648409 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-4f7628-5cc968cb6c-644x7\" (UID: \"d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f7628-5cc968cb6c-644x7" Oct 30 01:52:56.650603 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.648526 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-4f7628-5cc968cb6c-644x7\" (UID: \"d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f7628-5cc968cb6c-644x7" Oct 30 01:52:56.650603 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.648622 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df8ed3ef-12e5-40a7-9322-4b2a5f921d60-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-92f0cc-74f74b5d79-whvzj\" (UID: \"df8ed3ef-12e5-40a7-9322-4b2a5f921d60\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-92f0cc-74f74b5d79-whvzj" Oct 30 01:52:56.650603 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.648654 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf488\" (UniqueName: \"kubernetes.io/projected/d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2-kube-api-access-rf488\") pod \"maas-api-e2e-shared-b-w3-4f7628-5cc968cb6c-644x7\" (UID: \"d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f7628-5cc968cb6c-644x7" Oct 30 01:52:56.650603 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.648702 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5083acb-200e-4c21-8879-b7a1ebc8d337-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-92f0cc-7865694dd6-98f7j\" (UID: \"f5083acb-200e-4c21-8879-b7a1ebc8d337\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-92f0cc-7865694dd6-98f7j" Oct 30 01:52:56.651143 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.650759 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-4f7628-5cc968cb6c-644x7\" (UID: \"d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f7628-5cc968cb6c-644x7" Oct 30 01:52:56.695555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.654979 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-4f7628-5cc968cb6c-644x7\" (UID: \"d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f7628-5cc968cb6c-644x7" Oct 30 01:52:56.695555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.682618 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-4f7628-serving-cert\"" Oct 30 01:52:56.695555 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:52:56.684363 6221 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f7628-serving-cert: secret "maas-api-e2e-shared-b-w3-4f7628-serving-cert" not found Oct 30 01:52:56.695555 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:52:56.684455 6221 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2-maas-api-tls podName:d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2 nodeName:}" failed. No retries permitted until 2026-10-30 01:52:57.184432917 +0000 UTC m=+2100.762720231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2-maas-api-tls") pod "maas-api-e2e-shared-b-w3-4f7628-5cc968cb6c-644x7" (UID: "d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2") : secret "maas-api-e2e-shared-b-w3-4f7628-serving-cert" not found Oct 30 01:52:56.717583 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.711805 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf488\" (UniqueName: \"kubernetes.io/projected/d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2-kube-api-access-rf488\") pod \"maas-api-e2e-shared-b-w3-4f7628-5cc968cb6c-644x7\" (UID: \"d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f7628-5cc968cb6c-644x7" Oct 30 01:52:56.752017 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.751973 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df8ed3ef-12e5-40a7-9322-4b2a5f921d60-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-92f0cc-74f74b5d79-whvzj\" (UID: \"df8ed3ef-12e5-40a7-9322-4b2a5f921d60\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-92f0cc-74f74b5d79-whvzj" Oct 30 01:52:56.752598 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.752572 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5083acb-200e-4c21-8879-b7a1ebc8d337-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-92f0cc-7865694dd6-98f7j\" (UID: \"f5083acb-200e-4c21-8879-b7a1ebc8d337\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-92f0cc-7865694dd6-98f7j" Oct 30 01:52:56.753070 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.753047 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68jl9\" (UniqueName: \"kubernetes.io/projected/f5083acb-200e-4c21-8879-b7a1ebc8d337-kube-api-access-68jl9\") pod \"payload-processing-e2e-extmod-a-92f0cc-7865694dd6-98f7j\" (UID: \"f5083acb-200e-4c21-8879-b7a1ebc8d337\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-92f0cc-7865694dd6-98f7j" Oct 30 01:52:56.753234 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.753219 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89nwx\" (UniqueName: \"kubernetes.io/projected/df8ed3ef-12e5-40a7-9322-4b2a5f921d60-kube-api-access-89nwx\") pod \"payload-pre-processing-e2e-extmod-a-92f0cc-74f74b5d79-whvzj\" (UID: \"df8ed3ef-12e5-40a7-9322-4b2a5f921d60\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-92f0cc-74f74b5d79-whvzj" Oct 30 01:52:56.767554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.766047 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-92f0cc\"" Oct 30 01:52:56.774280 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.774210 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df8ed3ef-12e5-40a7-9322-4b2a5f921d60-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-92f0cc-74f74b5d79-whvzj\" (UID: \"df8ed3ef-12e5-40a7-9322-4b2a5f921d60\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-92f0cc-74f74b5d79-whvzj" Oct 30 01:52:56.774477 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.774358 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89nwx\" (UniqueName: \"kubernetes.io/projected/df8ed3ef-12e5-40a7-9322-4b2a5f921d60-kube-api-access-89nwx\") pod \"payload-pre-processing-e2e-extmod-a-92f0cc-74f74b5d79-whvzj\" (UID: \"df8ed3ef-12e5-40a7-9322-4b2a5f921d60\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-92f0cc-74f74b5d79-whvzj" Oct 30 01:52:56.775626 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.774978 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5083acb-200e-4c21-8879-b7a1ebc8d337-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-92f0cc-7865694dd6-98f7j\" (UID: \"f5083acb-200e-4c21-8879-b7a1ebc8d337\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-92f0cc-7865694dd6-98f7j" Oct 30 01:52:56.775626 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.775590 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68jl9\" (UniqueName: \"kubernetes.io/projected/f5083acb-200e-4c21-8879-b7a1ebc8d337-kube-api-access-68jl9\") pod \"payload-processing-e2e-extmod-a-92f0cc-7865694dd6-98f7j\" (UID: \"f5083acb-200e-4c21-8879-b7a1ebc8d337\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-92f0cc-7865694dd6-98f7j" Oct 30 01:52:56.832172 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.832142 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-92f0cc-dockercfg-z6n2x\"" Oct 30 01:52:56.838105 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.838077 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-92f0cc-74f74b5d79-whvzj" Oct 30 01:52:56.847817 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.847793 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2vsns_d332bbb4-c699-44e8-96ba-516d8ca03e95/ovn-acl-logging/0.log" Oct 30 01:52:56.877071 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:56.877047 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2vsns_d332bbb4-c699-44e8-96ba-516d8ca03e95/ovn-acl-logging/0.log" Oct 30 01:52:57.067359 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:57.067323 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-92f0cc-7865694dd6-98f7j" Oct 30 01:52:57.148931 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:57.148893 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-92f0cc-74f74b5d79-whvzj"] Oct 30 01:52:57.152987 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:52:57.152928 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf8ed3ef_12e5_40a7_9322_4b2a5f921d60.slice/crio-b6c8f7cb0b3d0694b709c36f84e43f9cfc821dbcdff550ef1ede9ad6d82a88bc WatchSource:0}: Error finding container b6c8f7cb0b3d0694b709c36f84e43f9cfc821dbcdff550ef1ede9ad6d82a88bc: Status 404 returned error can't find the container with id b6c8f7cb0b3d0694b709c36f84e43f9cfc821dbcdff550ef1ede9ad6d82a88bc Oct 30 01:52:57.264149 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:57.263500 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-4f7628-5cc968cb6c-644x7\" (UID: \"d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f7628-5cc968cb6c-644x7" Oct 30 01:52:57.267636 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:57.267606 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-4f7628-5cc968cb6c-644x7\" (UID: \"d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f7628-5cc968cb6c-644x7" Oct 30 01:52:57.302427 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:57.302389 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f7628-5cc968cb6c-644x7" Oct 30 01:52:57.374279 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:57.373017 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-92f0cc-7865694dd6-98f7j"] Oct 30 01:52:57.609425 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:57.609399 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f7628-5cc968cb6c-644x7"] Oct 30 01:52:57.611991 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:52:57.611968 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8e484a7_e60d_4ea4_ba8e_ddd19081c5e2.slice/crio-89be3ed96d6f87708cf580c88afb48cfbc7383a5c53c862dad2568d21d7ed426 WatchSource:0}: Error finding container 89be3ed96d6f87708cf580c88afb48cfbc7383a5c53c862dad2568d21d7ed426: Status 404 returned error can't find the container with id 89be3ed96d6f87708cf580c88afb48cfbc7383a5c53c862dad2568d21d7ed426 Oct 30 01:52:57.962075 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:57.962029 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f7628-5cc968cb6c-644x7" event={"ID":"d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2","Type":"ContainerStarted","Data":"89be3ed96d6f87708cf580c88afb48cfbc7383a5c53c862dad2568d21d7ed426"} Oct 30 01:52:57.963819 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:57.963792 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-92f0cc-74f74b5d79-whvzj" event={"ID":"df8ed3ef-12e5-40a7-9322-4b2a5f921d60","Type":"ContainerStarted","Data":"cf562c45bc3367cc5b5b9f65ba0179d308f863c140d26f9308800fa0440f4008"} Oct 30 01:52:57.963951 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:57.963835 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-92f0cc-74f74b5d79-whvzj" event={"ID":"df8ed3ef-12e5-40a7-9322-4b2a5f921d60","Type":"ContainerStarted","Data":"b6c8f7cb0b3d0694b709c36f84e43f9cfc821dbcdff550ef1ede9ad6d82a88bc"} Oct 30 01:52:57.963951 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:57.963896 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-92f0cc-74f74b5d79-whvzj" Oct 30 01:52:57.965451 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:57.965422 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-92f0cc-7865694dd6-98f7j" event={"ID":"f5083acb-200e-4c21-8879-b7a1ebc8d337","Type":"ContainerStarted","Data":"943ba0264d375885b278adba7da6ec92fd1700718522e9c1e51136cd4c8a35b0"} Oct 30 01:52:57.965576 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:57.965463 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-92f0cc-7865694dd6-98f7j" event={"ID":"f5083acb-200e-4c21-8879-b7a1ebc8d337","Type":"ContainerStarted","Data":"60b5061e48445bdc567227eca7db0cb0a8cbbfa70f4d6d46e44b397bd0baa6d4"} Oct 30 01:52:57.965576 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:57.965573 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-92f0cc-7865694dd6-98f7j" Oct 30 01:52:58.002758 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:58.002715 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-92f0cc-74f74b5d79-whvzj" podStartSLOduration=2.002703068 podStartE2EDuration="2.002703068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:52:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:52:58.001843267 +0000 UTC m=+2101.580130603" watchObservedRunningTime="2026-10-30 01:52:58.002703068 +0000 UTC m=+2101.580990405" Oct 30 01:52:58.042364 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:58.042304 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-92f0cc-7865694dd6-98f7j" podStartSLOduration=2.042291695 podStartE2EDuration="2.042291695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:52:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:52:58.040563274 +0000 UTC m=+2101.618850607" watchObservedRunningTime="2026-10-30 01:52:58.042291695 +0000 UTC m=+2101.620579031" Oct 30 01:52:58.973931 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:58.973887 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f7628-5cc968cb6c-644x7" event={"ID":"d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2","Type":"ContainerStarted","Data":"2cfd9a7fdd62ab716bbbc757598308f6ca969e3f01dc7d22b785931c61c7e3e4"} Oct 30 01:52:58.974420 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:58.974130 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f7628-5cc968cb6c-644x7" Oct 30 01:52:59.023582 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:52:59.023507 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f7628-5cc968cb6c-644x7" podStartSLOduration=2.6678288549999998 podStartE2EDuration="3.023492196s" totalImagesPullingTime="355.663341ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:52:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:52:59.019902491 +0000 UTC m=+2102.598189828" watchObservedRunningTime="2026-10-30 01:52:59.023492196 +0000 UTC m=+2102.601779533" Oct 30 01:53:03.523066 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:03.523010 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-861cd2/auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5" podUID="d001b9fc-56db-4ba1-8fcc-1c1e3af02e12" containerName="main" containerID="cri-o://cc66566cae7f2d999908d8c331e05e979b341771a1e5860023e1b18b94757724" gracePeriod=45 Oct 30 01:53:03.793044 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:03.793016 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-861cd2/auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5" Oct 30 01:53:03.865471 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:03.865422 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d001b9fc-56db-4ba1-8fcc-1c1e3af02e12-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d001b9fc-56db-4ba1-8fcc-1c1e3af02e12-dshm\") pod \"d001b9fc-56db-4ba1-8fcc-1c1e3af02e12\" (UID: \"d001b9fc-56db-4ba1-8fcc-1c1e3af02e12\") " Oct 30 01:53:03.865674 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:03.865626 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d001b9fc-56db-4ba1-8fcc-1c1e3af02e12-tls-certs\" (UniqueName: \"kubernetes.io/secret/d001b9fc-56db-4ba1-8fcc-1c1e3af02e12-tls-certs\") pod \"d001b9fc-56db-4ba1-8fcc-1c1e3af02e12\" (UID: \"d001b9fc-56db-4ba1-8fcc-1c1e3af02e12\") " Oct 30 01:53:03.865674 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:03.865661 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d001b9fc-56db-4ba1-8fcc-1c1e3af02e12-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d001b9fc-56db-4ba1-8fcc-1c1e3af02e12-tmp-dir\") pod \"d001b9fc-56db-4ba1-8fcc-1c1e3af02e12\" (UID: \"d001b9fc-56db-4ba1-8fcc-1c1e3af02e12\") " Oct 30 01:53:03.865831 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:03.865684 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d001b9fc-56db-4ba1-8fcc-1c1e3af02e12-home\" (UniqueName: \"kubernetes.io/empty-dir/d001b9fc-56db-4ba1-8fcc-1c1e3af02e12-home\") pod \"d001b9fc-56db-4ba1-8fcc-1c1e3af02e12\" (UID: \"d001b9fc-56db-4ba1-8fcc-1c1e3af02e12\") " Oct 30 01:53:03.865831 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:03.865791 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d001b9fc-56db-4ba1-8fcc-1c1e3af02e12-kube-api-access-rgvd6\" (UniqueName: \"kubernetes.io/projected/d001b9fc-56db-4ba1-8fcc-1c1e3af02e12-kube-api-access-rgvd6\") pod \"d001b9fc-56db-4ba1-8fcc-1c1e3af02e12\" (UID: \"d001b9fc-56db-4ba1-8fcc-1c1e3af02e12\") " Oct 30 01:53:03.865939 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:03.865830 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d001b9fc-56db-4ba1-8fcc-1c1e3af02e12-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d001b9fc-56db-4ba1-8fcc-1c1e3af02e12-model-cache\") pod \"d001b9fc-56db-4ba1-8fcc-1c1e3af02e12\" (UID: \"d001b9fc-56db-4ba1-8fcc-1c1e3af02e12\") " Oct 30 01:53:03.866282 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:03.866092 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d001b9fc-56db-4ba1-8fcc-1c1e3af02e12-home" pod "d001b9fc-56db-4ba1-8fcc-1c1e3af02e12" (UID: "d001b9fc-56db-4ba1-8fcc-1c1e3af02e12"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:53:03.866429 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:03.866376 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d001b9fc-56db-4ba1-8fcc-1c1e3af02e12-model-cache" pod "d001b9fc-56db-4ba1-8fcc-1c1e3af02e12" (UID: "d001b9fc-56db-4ba1-8fcc-1c1e3af02e12"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:53:03.866503 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:03.866432 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d001b9fc-56db-4ba1-8fcc-1c1e3af02e12-tmp-dir" pod "d001b9fc-56db-4ba1-8fcc-1c1e3af02e12" (UID: "d001b9fc-56db-4ba1-8fcc-1c1e3af02e12"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:53:03.866503 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:03.866472 6221 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d001b9fc-56db-4ba1-8fcc-1c1e3af02e12-home\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:53:03.866503 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:03.866494 6221 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d001b9fc-56db-4ba1-8fcc-1c1e3af02e12-model-cache\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:53:03.867949 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:03.867925 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d001b9fc-56db-4ba1-8fcc-1c1e3af02e12-dshm" pod "d001b9fc-56db-4ba1-8fcc-1c1e3af02e12" (UID: "d001b9fc-56db-4ba1-8fcc-1c1e3af02e12"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:53:03.868140 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:03.868118 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d001b9fc-56db-4ba1-8fcc-1c1e3af02e12-tls-certs" pod "d001b9fc-56db-4ba1-8fcc-1c1e3af02e12" (UID: "d001b9fc-56db-4ba1-8fcc-1c1e3af02e12"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:53:03.868595 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:03.868576 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d001b9fc-56db-4ba1-8fcc-1c1e3af02e12-kube-api-access-rgvd6" pod "d001b9fc-56db-4ba1-8fcc-1c1e3af02e12" (UID: "d001b9fc-56db-4ba1-8fcc-1c1e3af02e12"). InnerVolumeSpecName "kube-api-access-rgvd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:53:03.968122 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:03.968073 6221 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d001b9fc-56db-4ba1-8fcc-1c1e3af02e12-tls-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:53:03.968122 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:03.968111 6221 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d001b9fc-56db-4ba1-8fcc-1c1e3af02e12-tmp-dir\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:53:03.968122 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:03.968126 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgvd6\" (UniqueName: \"kubernetes.io/projected/d001b9fc-56db-4ba1-8fcc-1c1e3af02e12-kube-api-access-rgvd6\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:53:03.968371 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:03.968137 6221 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d001b9fc-56db-4ba1-8fcc-1c1e3af02e12-dshm\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:53:04.023301 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:04.023268 6221 generic.go:356] "Generic (PLEG): container finished" podID="d001b9fc-56db-4ba1-8fcc-1c1e3af02e12" containerID="cc66566cae7f2d999908d8c331e05e979b341771a1e5860023e1b18b94757724" exitCode=0 Oct 30 01:53:04.023481 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:04.023322 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-861cd2/auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5" Oct 30 01:53:04.023481 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:04.023354 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-861cd2/auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5" event={"ID":"d001b9fc-56db-4ba1-8fcc-1c1e3af02e12","Type":"ContainerDied","Data":"cc66566cae7f2d999908d8c331e05e979b341771a1e5860023e1b18b94757724"} Oct 30 01:53:04.023481 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:04.023393 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-861cd2/auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5" event={"ID":"d001b9fc-56db-4ba1-8fcc-1c1e3af02e12","Type":"ContainerDied","Data":"5bb7a4be5d01bc1cca666d28fff2ed0a1b4519fcd2db330b7e3096f5bc5b1397"} Oct 30 01:53:04.023481 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:04.023417 6221 scope.go:122] "RemoveContainer" containerID="cc66566cae7f2d999908d8c331e05e979b341771a1e5860023e1b18b94757724" Oct 30 01:53:04.035938 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:04.035842 6221 scope.go:122] "RemoveContainer" containerID="cc66566cae7f2d999908d8c331e05e979b341771a1e5860023e1b18b94757724" Oct 30 01:53:04.036265 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:53:04.036238 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc66566cae7f2d999908d8c331e05e979b341771a1e5860023e1b18b94757724\": container with ID starting with cc66566cae7f2d999908d8c331e05e979b341771a1e5860023e1b18b94757724 not found: ID does not exist" containerID="cc66566cae7f2d999908d8c331e05e979b341771a1e5860023e1b18b94757724" Oct 30 01:53:04.036366 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:04.036268 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc66566cae7f2d999908d8c331e05e979b341771a1e5860023e1b18b94757724"} err="failed to get container status \"cc66566cae7f2d999908d8c331e05e979b341771a1e5860023e1b18b94757724\": rpc error: code = NotFound desc = could not find container \"cc66566cae7f2d999908d8c331e05e979b341771a1e5860023e1b18b94757724\": container with ID starting with cc66566cae7f2d999908d8c331e05e979b341771a1e5860023e1b18b94757724 not found: ID does not exist" Oct 30 01:53:04.077333 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:04.077245 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-861cd2/auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5"] Oct 30 01:53:04.080938 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:04.080908 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-861cd2/auth-test-model-861cd2-kserve-7d9d6bdd49-l5ft5"] Oct 30 01:53:04.548958 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:04.548925 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d001b9fc-56db-4ba1-8fcc-1c1e3af02e12" path="/var/lib/kubelet/pods/d001b9fc-56db-4ba1-8fcc-1c1e3af02e12/volumes" Oct 30 01:53:05.001859 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:05.001787 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f7628-5cc968cb6c-644x7" Oct 30 01:53:08.977377 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:08.977345 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-92f0cc-74f74b5d79-whvzj" Oct 30 01:53:08.977901 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:08.977706 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-92f0cc-7865694dd6-98f7j" Oct 30 01:53:18.991315 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:18.990930 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:53:19.776748 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:19.776715 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2e618f-f475b8c6c-lr97t"] Oct 30 01:53:19.777415 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:19.777400 6221 cpu_manager.go:414] "containerMap: removing container" podUID="d001b9fc-56db-4ba1-8fcc-1c1e3af02e12" containerName="main" Oct 30 01:53:19.777469 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:19.777416 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d001b9fc-56db-4ba1-8fcc-1c1e3af02e12" containerName="main" Oct 30 01:53:19.777540 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:19.777518 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="d001b9fc-56db-4ba1-8fcc-1c1e3af02e12" containerName="main" Oct 30 01:53:19.782544 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:19.782516 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2e618f-f475b8c6c-lr97t" Oct 30 01:53:19.798674 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:19.798645 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2e618f-f475b8c6c-lr97t"] Oct 30 01:53:19.807278 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:19.807254 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-2e618f\"" Oct 30 01:53:19.814712 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:19.814687 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zjbx\" (UniqueName: \"kubernetes.io/projected/fd5ba5d9-23ee-42d4-97f0-3224353b90ec-kube-api-access-7zjbx\") pod \"payload-pre-processing-e2e-extmod-b-2e618f-f475b8c6c-lr97t\" (UID: \"fd5ba5d9-23ee-42d4-97f0-3224353b90ec\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2e618f-f475b8c6c-lr97t" Oct 30 01:53:19.814991 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:19.814970 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd5ba5d9-23ee-42d4-97f0-3224353b90ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-2e618f-f475b8c6c-lr97t\" (UID: \"fd5ba5d9-23ee-42d4-97f0-3224353b90ec\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2e618f-f475b8c6c-lr97t" Oct 30 01:53:19.873235 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:19.873186 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2e618f-7975b944f8-8f6dz"] Oct 30 01:53:19.900905 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:19.900865 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-2e618f-dockercfg-qlhr2\"" Oct 30 01:53:19.911190 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:19.911159 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2e618f-7975b944f8-8f6dz"] Oct 30 01:53:19.911602 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:19.911491 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-2e618f-7975b944f8-8f6dz" Oct 30 01:53:19.917851 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:19.917823 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd5ba5d9-23ee-42d4-97f0-3224353b90ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-2e618f-f475b8c6c-lr97t\" (UID: \"fd5ba5d9-23ee-42d4-97f0-3224353b90ec\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2e618f-f475b8c6c-lr97t" Oct 30 01:53:19.918115 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:19.918093 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zjbx\" (UniqueName: \"kubernetes.io/projected/fd5ba5d9-23ee-42d4-97f0-3224353b90ec-kube-api-access-7zjbx\") pod \"payload-pre-processing-e2e-extmod-b-2e618f-f475b8c6c-lr97t\" (UID: \"fd5ba5d9-23ee-42d4-97f0-3224353b90ec\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2e618f-f475b8c6c-lr97t" Oct 30 01:53:19.933200 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:19.933159 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd5ba5d9-23ee-42d4-97f0-3224353b90ec-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-2e618f-f475b8c6c-lr97t\" (UID: \"fd5ba5d9-23ee-42d4-97f0-3224353b90ec\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2e618f-f475b8c6c-lr97t" Oct 30 01:53:20.024438 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:20.024388 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwx2j\" (UniqueName: \"kubernetes.io/projected/16bdf204-496b-4e72-b5cd-e7b5719043e5-kube-api-access-vwx2j\") pod \"payload-processing-e2e-extmod-b-2e618f-7975b944f8-8f6dz\" (UID: \"16bdf204-496b-4e72-b5cd-e7b5719043e5\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2e618f-7975b944f8-8f6dz" Oct 30 01:53:20.025118 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:20.025093 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16bdf204-496b-4e72-b5cd-e7b5719043e5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-2e618f-7975b944f8-8f6dz\" (UID: \"16bdf204-496b-4e72-b5cd-e7b5719043e5\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2e618f-7975b944f8-8f6dz" Oct 30 01:53:20.030601 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:20.030520 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zjbx\" (UniqueName: \"kubernetes.io/projected/fd5ba5d9-23ee-42d4-97f0-3224353b90ec-kube-api-access-7zjbx\") pod \"payload-pre-processing-e2e-extmod-b-2e618f-f475b8c6c-lr97t\" (UID: \"fd5ba5d9-23ee-42d4-97f0-3224353b90ec\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2e618f-f475b8c6c-lr97t" Oct 30 01:53:20.096672 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:20.096635 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2e618f-f475b8c6c-lr97t" Oct 30 01:53:20.127653 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:20.127612 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwx2j\" (UniqueName: \"kubernetes.io/projected/16bdf204-496b-4e72-b5cd-e7b5719043e5-kube-api-access-vwx2j\") pod \"payload-processing-e2e-extmod-b-2e618f-7975b944f8-8f6dz\" (UID: \"16bdf204-496b-4e72-b5cd-e7b5719043e5\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2e618f-7975b944f8-8f6dz" Oct 30 01:53:20.127812 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:20.127671 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16bdf204-496b-4e72-b5cd-e7b5719043e5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-2e618f-7975b944f8-8f6dz\" (UID: \"16bdf204-496b-4e72-b5cd-e7b5719043e5\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2e618f-7975b944f8-8f6dz" Oct 30 01:53:20.128400 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:20.128378 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16bdf204-496b-4e72-b5cd-e7b5719043e5-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-2e618f-7975b944f8-8f6dz\" (UID: \"16bdf204-496b-4e72-b5cd-e7b5719043e5\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2e618f-7975b944f8-8f6dz" Oct 30 01:53:20.149286 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:20.149253 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwx2j\" (UniqueName: \"kubernetes.io/projected/16bdf204-496b-4e72-b5cd-e7b5719043e5-kube-api-access-vwx2j\") pod \"payload-processing-e2e-extmod-b-2e618f-7975b944f8-8f6dz\" (UID: \"16bdf204-496b-4e72-b5cd-e7b5719043e5\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2e618f-7975b944f8-8f6dz" Oct 30 01:53:20.265154 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:20.265120 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-2e618f-7975b944f8-8f6dz" Oct 30 01:53:20.408433 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:20.408392 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2e618f-f475b8c6c-lr97t"] Oct 30 01:53:20.409391 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:53:20.409365 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd5ba5d9_23ee_42d4_97f0_3224353b90ec.slice/crio-c7fe03d12dadd73019bc6f8da2b68919e6b42e6152a8cf3fc0bf9526fd5a9e5e WatchSource:0}: Error finding container c7fe03d12dadd73019bc6f8da2b68919e6b42e6152a8cf3fc0bf9526fd5a9e5e: Status 404 returned error can't find the container with id c7fe03d12dadd73019bc6f8da2b68919e6b42e6152a8cf3fc0bf9526fd5a9e5e Oct 30 01:53:20.571388 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:20.571365 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2e618f-7975b944f8-8f6dz"] Oct 30 01:53:20.573408 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:53:20.573379 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16bdf204_496b_4e72_b5cd_e7b5719043e5.slice/crio-a76b137db87dece91da6800f32a1aab90ff79811ddac48250c8b2287b1063869 WatchSource:0}: Error finding container a76b137db87dece91da6800f32a1aab90ff79811ddac48250c8b2287b1063869: Status 404 returned error can't find the container with id a76b137db87dece91da6800f32a1aab90ff79811ddac48250c8b2287b1063869 Oct 30 01:53:21.168354 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:21.168317 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-2e618f-7975b944f8-8f6dz" event={"ID":"16bdf204-496b-4e72-b5cd-e7b5719043e5","Type":"ContainerStarted","Data":"f63350ef310c4c3a1709316f216f429d91a63cfaada4102d24d41133770042a5"} Oct 30 01:53:21.168836 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:21.168382 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-2e618f-7975b944f8-8f6dz" event={"ID":"16bdf204-496b-4e72-b5cd-e7b5719043e5","Type":"ContainerStarted","Data":"a76b137db87dece91da6800f32a1aab90ff79811ddac48250c8b2287b1063869"} Oct 30 01:53:21.168836 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:21.168437 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-2e618f-7975b944f8-8f6dz" Oct 30 01:53:21.170060 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:21.170038 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2e618f-f475b8c6c-lr97t" event={"ID":"fd5ba5d9-23ee-42d4-97f0-3224353b90ec","Type":"ContainerStarted","Data":"a161132d7da7853c90ed85354f27f2bd7f2ad298019d4667560d3e7bcbdd7651"} Oct 30 01:53:21.170179 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:21.170073 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2e618f-f475b8c6c-lr97t" event={"ID":"fd5ba5d9-23ee-42d4-97f0-3224353b90ec","Type":"ContainerStarted","Data":"c7fe03d12dadd73019bc6f8da2b68919e6b42e6152a8cf3fc0bf9526fd5a9e5e"} Oct 30 01:53:21.170179 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:21.170145 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2e618f-f475b8c6c-lr97t" Oct 30 01:53:21.207863 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:21.207814 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-2e618f-7975b944f8-8f6dz" podStartSLOduration=2.207799863 podStartE2EDuration="2.207799863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:53:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:53:21.206051293 +0000 UTC m=+2124.784338629" watchObservedRunningTime="2026-10-30 01:53:21.207799863 +0000 UTC m=+2124.786087196" Oct 30 01:53:21.247605 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:21.247555 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2e618f-f475b8c6c-lr97t" podStartSLOduration=2.247517716 podStartE2EDuration="2.247517716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:53:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:53:21.244761413 +0000 UTC m=+2124.823048750" watchObservedRunningTime="2026-10-30 01:53:21.247517716 +0000 UTC m=+2124.825805052" Oct 30 01:53:21.493345 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:53:21.493265 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:53:25.630158 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:25.630123 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-b6b7fc/auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp"] Oct 30 01:53:32.217397 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:32.217365 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2e618f-f475b8c6c-lr97t" Oct 30 01:53:32.217867 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:32.217430 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-2e618f-7975b944f8-8f6dz" Oct 30 01:53:39.608363 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:39.608296 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7492-bdc0f5"] Oct 30 01:53:39.614270 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:39.614244 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7492-bdc0f5" Oct 30 01:53:39.626619 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:39.626582 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7492-bdc0f5"] Oct 30 01:53:39.633267 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:39.633238 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-pm9kh\"" Oct 30 01:53:39.757810 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:39.757770 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqdqc\" (UniqueName: \"kubernetes.io/projected/88ed0eb1-f48a-4fe2-8bd3-9d2472ccdb2e-kube-api-access-hqdqc\") pod \"test-curl-7492-bdc0f5\" (UID: \"88ed0eb1-f48a-4fe2-8bd3-9d2472ccdb2e\") " pod="opendatahub/test-curl-7492-bdc0f5" Oct 30 01:53:39.859599 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:39.859477 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqdqc\" (UniqueName: \"kubernetes.io/projected/88ed0eb1-f48a-4fe2-8bd3-9d2472ccdb2e-kube-api-access-hqdqc\") pod \"test-curl-7492-bdc0f5\" (UID: \"88ed0eb1-f48a-4fe2-8bd3-9d2472ccdb2e\") " pod="opendatahub/test-curl-7492-bdc0f5" Oct 30 01:53:39.879483 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:39.879447 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqdqc\" (UniqueName: \"kubernetes.io/projected/88ed0eb1-f48a-4fe2-8bd3-9d2472ccdb2e-kube-api-access-hqdqc\") pod \"test-curl-7492-bdc0f5\" (UID: \"88ed0eb1-f48a-4fe2-8bd3-9d2472ccdb2e\") " pod="opendatahub/test-curl-7492-bdc0f5" Oct 30 01:53:39.937857 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:39.937824 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7492-bdc0f5" Oct 30 01:53:40.250149 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:40.250122 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7492-bdc0f5"] Oct 30 01:53:40.252965 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:53:40.252930 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88ed0eb1_f48a_4fe2_8bd3_9d2472ccdb2e.slice/crio-d4ee22a8a900a9886b68b651e032ed6e991dcd3470c82ab0d9d47ffa71759e8f WatchSource:0}: Error finding container d4ee22a8a900a9886b68b651e032ed6e991dcd3470c82ab0d9d47ffa71759e8f: Status 404 returned error can't find the container with id d4ee22a8a900a9886b68b651e032ed6e991dcd3470c82ab0d9d47ffa71759e8f Oct 30 01:53:40.438233 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:40.438186 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7492-bdc0f5" event={"ID":"88ed0eb1-f48a-4fe2-8bd3-9d2472ccdb2e","Type":"ContainerStarted","Data":"d4ee22a8a900a9886b68b651e032ed6e991dcd3470c82ab0d9d47ffa71759e8f"} Oct 30 01:53:40.651681 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:40.651637 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-b6b7fc/auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp" podUID="0794883c-76c6-4a6f-8f12-6e65420ef583" containerName="main" containerID="cri-o://fd343432d731d1f3615c8f72c3e9cf0a270f19d6bf0552e7fd004ec57b35ba9a" gracePeriod=45 Oct 30 01:53:40.759085 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:40.759051 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-b6b7fc/auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp" podUID="0794883c-76c6-4a6f-8f12-6e65420ef583" containerName="main" probeResult="failure" output="Get \"https://10.134.0.158:8000/ready\": dial tcp 10.134.0.158:8000: connect: connection refused" Oct 30 01:53:40.923984 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:40.923960 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-b6b7fc/auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp" Oct 30 01:53:41.084557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:41.083970 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0794883c-76c6-4a6f-8f12-6e65420ef583-tls-certs\" (UniqueName: \"kubernetes.io/secret/0794883c-76c6-4a6f-8f12-6e65420ef583-tls-certs\") pod \"0794883c-76c6-4a6f-8f12-6e65420ef583\" (UID: \"0794883c-76c6-4a6f-8f12-6e65420ef583\") " Oct 30 01:53:41.084557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:41.084134 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0794883c-76c6-4a6f-8f12-6e65420ef583-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0794883c-76c6-4a6f-8f12-6e65420ef583-tmp-dir\") pod \"0794883c-76c6-4a6f-8f12-6e65420ef583\" (UID: \"0794883c-76c6-4a6f-8f12-6e65420ef583\") " Oct 30 01:53:41.084557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:41.084186 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0794883c-76c6-4a6f-8f12-6e65420ef583-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0794883c-76c6-4a6f-8f12-6e65420ef583-model-cache\") pod \"0794883c-76c6-4a6f-8f12-6e65420ef583\" (UID: \"0794883c-76c6-4a6f-8f12-6e65420ef583\") " Oct 30 01:53:41.084557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:41.084314 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0794883c-76c6-4a6f-8f12-6e65420ef583-home\" (UniqueName: \"kubernetes.io/empty-dir/0794883c-76c6-4a6f-8f12-6e65420ef583-home\") pod \"0794883c-76c6-4a6f-8f12-6e65420ef583\" (UID: \"0794883c-76c6-4a6f-8f12-6e65420ef583\") " Oct 30 01:53:41.084557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:41.084351 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0794883c-76c6-4a6f-8f12-6e65420ef583-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0794883c-76c6-4a6f-8f12-6e65420ef583-dshm\") pod \"0794883c-76c6-4a6f-8f12-6e65420ef583\" (UID: \"0794883c-76c6-4a6f-8f12-6e65420ef583\") " Oct 30 01:53:41.084557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:41.084475 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0794883c-76c6-4a6f-8f12-6e65420ef583-kube-api-access-7z2ks\" (UniqueName: \"kubernetes.io/projected/0794883c-76c6-4a6f-8f12-6e65420ef583-kube-api-access-7z2ks\") pod \"0794883c-76c6-4a6f-8f12-6e65420ef583\" (UID: \"0794883c-76c6-4a6f-8f12-6e65420ef583\") " Oct 30 01:53:41.104583 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:41.088628 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0794883c-76c6-4a6f-8f12-6e65420ef583-model-cache" pod "0794883c-76c6-4a6f-8f12-6e65420ef583" (UID: "0794883c-76c6-4a6f-8f12-6e65420ef583"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:53:41.104583 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:41.090511 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0794883c-76c6-4a6f-8f12-6e65420ef583-home" pod "0794883c-76c6-4a6f-8f12-6e65420ef583" (UID: "0794883c-76c6-4a6f-8f12-6e65420ef583"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:53:41.104583 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:41.091613 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0794883c-76c6-4a6f-8f12-6e65420ef583-tmp-dir" pod "0794883c-76c6-4a6f-8f12-6e65420ef583" (UID: "0794883c-76c6-4a6f-8f12-6e65420ef583"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:53:41.119173 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:41.117743 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0794883c-76c6-4a6f-8f12-6e65420ef583-dshm" pod "0794883c-76c6-4a6f-8f12-6e65420ef583" (UID: "0794883c-76c6-4a6f-8f12-6e65420ef583"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:53:41.167467 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:41.166952 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0794883c-76c6-4a6f-8f12-6e65420ef583-kube-api-access-7z2ks" pod "0794883c-76c6-4a6f-8f12-6e65420ef583" (UID: "0794883c-76c6-4a6f-8f12-6e65420ef583"). InnerVolumeSpecName "kube-api-access-7z2ks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:53:41.178589 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:41.176222 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0794883c-76c6-4a6f-8f12-6e65420ef583-tls-certs" pod "0794883c-76c6-4a6f-8f12-6e65420ef583" (UID: "0794883c-76c6-4a6f-8f12-6e65420ef583"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:53:41.187018 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:41.186991 6221 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0794883c-76c6-4a6f-8f12-6e65420ef583-tmp-dir\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:53:41.187161 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:41.187022 6221 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0794883c-76c6-4a6f-8f12-6e65420ef583-model-cache\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:53:41.187161 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:41.187036 6221 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0794883c-76c6-4a6f-8f12-6e65420ef583-home\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:53:41.187161 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:41.187046 6221 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0794883c-76c6-4a6f-8f12-6e65420ef583-dshm\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:53:41.187161 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:41.187054 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z2ks\" (UniqueName: \"kubernetes.io/projected/0794883c-76c6-4a6f-8f12-6e65420ef583-kube-api-access-7z2ks\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:53:41.187161 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:41.187062 6221 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0794883c-76c6-4a6f-8f12-6e65420ef583-tls-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:53:41.584765 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:41.584321 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7492-bdc0f5" event={"ID":"88ed0eb1-f48a-4fe2-8bd3-9d2472ccdb2e","Type":"ContainerStarted","Data":"1e2e9a0bb54eec807b121fc992d51464fef24ba9d16a761c48fb2c1a09599967"} Oct 30 01:53:41.586304 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:41.586269 6221 generic.go:356] "Generic (PLEG): container finished" podID="0794883c-76c6-4a6f-8f12-6e65420ef583" containerID="fd343432d731d1f3615c8f72c3e9cf0a270f19d6bf0552e7fd004ec57b35ba9a" exitCode=0 Oct 30 01:53:41.586628 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:41.586360 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-b6b7fc/auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp" Oct 30 01:53:41.586628 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:41.586384 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-b6b7fc/auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp" event={"ID":"0794883c-76c6-4a6f-8f12-6e65420ef583","Type":"ContainerDied","Data":"fd343432d731d1f3615c8f72c3e9cf0a270f19d6bf0552e7fd004ec57b35ba9a"} Oct 30 01:53:41.586628 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:41.586415 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-b6b7fc/auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp" event={"ID":"0794883c-76c6-4a6f-8f12-6e65420ef583","Type":"ContainerDied","Data":"97cd20fcd98f07289ebfafbfa818effa2ada49e063321ca70da745aaa442f59d"} Oct 30 01:53:41.586628 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:41.586439 6221 scope.go:122] "RemoveContainer" containerID="fd343432d731d1f3615c8f72c3e9cf0a270f19d6bf0552e7fd004ec57b35ba9a" Oct 30 01:53:41.607705 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:41.607679 6221 scope.go:122] "RemoveContainer" containerID="fd343432d731d1f3615c8f72c3e9cf0a270f19d6bf0552e7fd004ec57b35ba9a" Oct 30 01:53:41.608113 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:53:41.608046 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd343432d731d1f3615c8f72c3e9cf0a270f19d6bf0552e7fd004ec57b35ba9a\": container with ID starting with fd343432d731d1f3615c8f72c3e9cf0a270f19d6bf0552e7fd004ec57b35ba9a not found: ID does not exist" containerID="fd343432d731d1f3615c8f72c3e9cf0a270f19d6bf0552e7fd004ec57b35ba9a" Oct 30 01:53:41.608113 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:41.608087 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd343432d731d1f3615c8f72c3e9cf0a270f19d6bf0552e7fd004ec57b35ba9a"} err="failed to get container status \"fd343432d731d1f3615c8f72c3e9cf0a270f19d6bf0552e7fd004ec57b35ba9a\": rpc error: code = NotFound desc = could not find container \"fd343432d731d1f3615c8f72c3e9cf0a270f19d6bf0552e7fd004ec57b35ba9a\": container with ID starting with fd343432d731d1f3615c8f72c3e9cf0a270f19d6bf0552e7fd004ec57b35ba9a not found: ID does not exist" Oct 30 01:53:41.629296 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:41.629232 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7492-bdc0f5" podStartSLOduration=2.6292140059999998 podStartE2EDuration="2.629214006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:53:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:53:41.625423493 +0000 UTC m=+2145.203710842" watchObservedRunningTime="2026-10-30 01:53:41.629214006 +0000 UTC m=+2145.207501342" Oct 30 01:53:41.667817 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:53:41.667773 6221 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0794883c_76c6_4a6f_8f12_6e65420ef583.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0794883c_76c6_4a6f_8f12_6e65420ef583.slice/crio-97cd20fcd98f07289ebfafbfa818effa2ada49e063321ca70da745aaa442f59d\": RecentStats: unable to find data in memory cache]" Oct 30 01:53:41.668385 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:41.667959 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-b6b7fc/auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp"] Oct 30 01:53:41.678519 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:41.678483 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-b6b7fc/auth-test-model-b6b7fc-kserve-5c8ff696f-42nvp"] Oct 30 01:53:42.525576 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:42.525518 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7492-bdc0f5"] Oct 30 01:53:42.531309 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:42.531280 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7492-bdc0f5"] Oct 30 01:53:42.549254 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:42.549223 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0794883c-76c6-4a6f-8f12-6e65420ef583" path="/var/lib/kubelet/pods/0794883c-76c6-4a6f-8f12-6e65420ef583/volumes" Oct 30 01:53:43.604478 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:43.604418 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7492-bdc0f5" podUID="88ed0eb1-f48a-4fe2-8bd3-9d2472ccdb2e" containerName="test-curl-7492-bdc0f5" containerID="cri-o://1e2e9a0bb54eec807b121fc992d51464fef24ba9d16a761c48fb2c1a09599967" gracePeriod=2 Oct 30 01:53:44.685553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:44.670860 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr"] Oct 30 01:53:44.694638 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:44.687005 6221 cpu_manager.go:414] "containerMap: removing container" podUID="88ed0eb1-f48a-4fe2-8bd3-9d2472ccdb2e" containerName="test-curl-7492-bdc0f5" Oct 30 01:53:44.694975 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:44.694947 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88ed0eb1-f48a-4fe2-8bd3-9d2472ccdb2e" containerName="test-curl-7492-bdc0f5" Oct 30 01:53:44.695268 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:44.695253 6221 cpu_manager.go:414] "containerMap: removing container" podUID="0794883c-76c6-4a6f-8f12-6e65420ef583" containerName="main" Oct 30 01:53:44.695364 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:44.695353 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0794883c-76c6-4a6f-8f12-6e65420ef583" containerName="main" Oct 30 01:53:44.695732 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:44.695716 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="0794883c-76c6-4a6f-8f12-6e65420ef583" containerName="main" Oct 30 01:53:44.697120 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:44.696872 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="88ed0eb1-f48a-4fe2-8bd3-9d2472ccdb2e" containerName="test-curl-7492-bdc0f5" Oct 30 01:53:44.704388 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:44.704369 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" Oct 30 01:53:44.730579 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:44.729902 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr"] Oct 30 01:53:44.730579 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:44.730484 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-2eae39-openshift-default-dockercfg-hzn9m\"" Oct 30 01:53:44.780563 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:44.774108 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcd89198-b4a1-40cd-9850-3dda3af7bbab-credential-socket\") pod \"e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr\" (UID: \"dcd89198-b4a1-40cd-9850-3dda3af7bbab\") " pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" Oct 30 01:53:44.780563 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:44.774227 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcd89198-b4a1-40cd-9850-3dda3af7bbab-istio-token\") pod \"e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr\" (UID: \"dcd89198-b4a1-40cd-9850-3dda3af7bbab\") " pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" Oct 30 01:53:44.780563 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:44.774280 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcd89198-b4a1-40cd-9850-3dda3af7bbab-istio-envoy\") pod \"e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr\" (UID: \"dcd89198-b4a1-40cd-9850-3dda3af7bbab\") " pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" Oct 30 01:53:44.780563 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:44.774325 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcd89198-b4a1-40cd-9850-3dda3af7bbab-istio-podinfo\") pod \"e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr\" (UID: \"dcd89198-b4a1-40cd-9850-3dda3af7bbab\") " pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" Oct 30 01:53:44.780563 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:44.774393 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcd89198-b4a1-40cd-9850-3dda3af7bbab-workload-certs\") pod \"e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr\" (UID: \"dcd89198-b4a1-40cd-9850-3dda3af7bbab\") " pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" Oct 30 01:53:44.780563 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:44.774450 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcd89198-b4a1-40cd-9850-3dda3af7bbab-istiod-ca-cert\") pod \"e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr\" (UID: \"dcd89198-b4a1-40cd-9850-3dda3af7bbab\") " pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" Oct 30 01:53:44.780563 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:44.774482 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcd89198-b4a1-40cd-9850-3dda3af7bbab-workload-socket\") pod \"e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr\" (UID: \"dcd89198-b4a1-40cd-9850-3dda3af7bbab\") " pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" Oct 30 01:53:44.780563 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:44.774636 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcd89198-b4a1-40cd-9850-3dda3af7bbab-istio-data\") pod \"e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr\" (UID: \"dcd89198-b4a1-40cd-9850-3dda3af7bbab\") " pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" Oct 30 01:53:44.780563 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:44.774681 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vzbx\" (UniqueName: \"kubernetes.io/projected/dcd89198-b4a1-40cd-9850-3dda3af7bbab-kube-api-access-9vzbx\") pod \"e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr\" (UID: \"dcd89198-b4a1-40cd-9850-3dda3af7bbab\") " pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" Oct 30 01:53:44.876170 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:44.876131 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcd89198-b4a1-40cd-9850-3dda3af7bbab-istio-envoy\") pod \"e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr\" (UID: \"dcd89198-b4a1-40cd-9850-3dda3af7bbab\") " pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" Oct 30 01:53:44.876170 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:44.876180 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcd89198-b4a1-40cd-9850-3dda3af7bbab-istio-podinfo\") pod \"e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr\" (UID: \"dcd89198-b4a1-40cd-9850-3dda3af7bbab\") " pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" Oct 30 01:53:44.876441 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:44.876222 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcd89198-b4a1-40cd-9850-3dda3af7bbab-workload-certs\") pod \"e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr\" (UID: \"dcd89198-b4a1-40cd-9850-3dda3af7bbab\") " pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" Oct 30 01:53:44.876441 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:44.876256 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcd89198-b4a1-40cd-9850-3dda3af7bbab-istiod-ca-cert\") pod \"e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr\" (UID: \"dcd89198-b4a1-40cd-9850-3dda3af7bbab\") " pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" Oct 30 01:53:44.876441 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:44.876277 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcd89198-b4a1-40cd-9850-3dda3af7bbab-workload-socket\") pod \"e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr\" (UID: \"dcd89198-b4a1-40cd-9850-3dda3af7bbab\") " pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" Oct 30 01:53:44.876441 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:44.876355 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcd89198-b4a1-40cd-9850-3dda3af7bbab-istio-data\") pod \"e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr\" (UID: \"dcd89198-b4a1-40cd-9850-3dda3af7bbab\") " pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" Oct 30 01:53:44.876441 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:44.876395 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vzbx\" (UniqueName: \"kubernetes.io/projected/dcd89198-b4a1-40cd-9850-3dda3af7bbab-kube-api-access-9vzbx\") pod \"e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr\" (UID: \"dcd89198-b4a1-40cd-9850-3dda3af7bbab\") " pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" Oct 30 01:53:44.876746 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:44.876466 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcd89198-b4a1-40cd-9850-3dda3af7bbab-credential-socket\") pod \"e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr\" (UID: \"dcd89198-b4a1-40cd-9850-3dda3af7bbab\") " pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" Oct 30 01:53:44.876746 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:44.876610 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcd89198-b4a1-40cd-9850-3dda3af7bbab-istio-token\") pod \"e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr\" (UID: \"dcd89198-b4a1-40cd-9850-3dda3af7bbab\") " pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" Oct 30 01:53:44.876746 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:44.876655 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcd89198-b4a1-40cd-9850-3dda3af7bbab-workload-certs\") pod \"e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr\" (UID: \"dcd89198-b4a1-40cd-9850-3dda3af7bbab\") " pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" Oct 30 01:53:44.876746 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:44.876706 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcd89198-b4a1-40cd-9850-3dda3af7bbab-workload-socket\") pod \"e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr\" (UID: \"dcd89198-b4a1-40cd-9850-3dda3af7bbab\") " pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" Oct 30 01:53:44.876956 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:44.876897 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcd89198-b4a1-40cd-9850-3dda3af7bbab-credential-socket\") pod \"e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr\" (UID: \"dcd89198-b4a1-40cd-9850-3dda3af7bbab\") " pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" Oct 30 01:53:44.877222 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:44.877203 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcd89198-b4a1-40cd-9850-3dda3af7bbab-istiod-ca-cert\") pod \"e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr\" (UID: \"dcd89198-b4a1-40cd-9850-3dda3af7bbab\") " pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" Oct 30 01:53:44.877394 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:44.877377 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcd89198-b4a1-40cd-9850-3dda3af7bbab-istio-data\") pod \"e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr\" (UID: \"dcd89198-b4a1-40cd-9850-3dda3af7bbab\") " pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" Oct 30 01:53:44.878838 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:44.878813 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcd89198-b4a1-40cd-9850-3dda3af7bbab-istio-envoy\") pod \"e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr\" (UID: \"dcd89198-b4a1-40cd-9850-3dda3af7bbab\") " pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" Oct 30 01:53:44.879615 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:44.879593 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcd89198-b4a1-40cd-9850-3dda3af7bbab-istio-podinfo\") pod \"e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr\" (UID: \"dcd89198-b4a1-40cd-9850-3dda3af7bbab\") " pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" Oct 30 01:53:44.896937 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:44.896896 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcd89198-b4a1-40cd-9850-3dda3af7bbab-istio-token\") pod \"e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr\" (UID: \"dcd89198-b4a1-40cd-9850-3dda3af7bbab\") " pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" Oct 30 01:53:44.897792 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:44.897736 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vzbx\" (UniqueName: \"kubernetes.io/projected/dcd89198-b4a1-40cd-9850-3dda3af7bbab-kube-api-access-9vzbx\") pod \"e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr\" (UID: \"dcd89198-b4a1-40cd-9850-3dda3af7bbab\") " pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" Oct 30 01:53:45.036458 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:45.036432 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" Oct 30 01:53:45.384501 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:45.384454 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr"] Oct 30 01:53:45.387176 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:53:45.387150 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcd89198_b4a1_40cd_9850_3dda3af7bbab.slice/crio-47da10f24f2938ec92cbc9e3b732dccf31d16773aea16c5aefca012f51a2023b WatchSource:0}: Error finding container 47da10f24f2938ec92cbc9e3b732dccf31d16773aea16c5aefca012f51a2023b: Status 404 returned error can't find the container with id 47da10f24f2938ec92cbc9e3b732dccf31d16773aea16c5aefca012f51a2023b Oct 30 01:53:45.389712 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:45.389679 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:53:45.389814 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:45.389743 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:53:45.389814 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:45.389773 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:53:45.666434 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:45.666402 6221 generic.go:356] "Generic (PLEG): container finished" podID="88ed0eb1-f48a-4fe2-8bd3-9d2472ccdb2e" containerID="1e2e9a0bb54eec807b121fc992d51464fef24ba9d16a761c48fb2c1a09599967" exitCode=137 Oct 30 01:53:45.668554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:45.668512 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" event={"ID":"dcd89198-b4a1-40cd-9850-3dda3af7bbab","Type":"ContainerStarted","Data":"aa3a19507a0829c2b275082658f45f955b9aa27e73a1190ae70ece556cb20d8c"} Oct 30 01:53:45.668707 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:45.668582 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" event={"ID":"dcd89198-b4a1-40cd-9850-3dda3af7bbab","Type":"ContainerStarted","Data":"47da10f24f2938ec92cbc9e3b732dccf31d16773aea16c5aefca012f51a2023b"} Oct 30 01:53:45.720061 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:45.720008 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" podStartSLOduration=1.7199939180000001 podStartE2EDuration="1.719993918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:53:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:53:45.715583385 +0000 UTC m=+2149.293870720" watchObservedRunningTime="2026-10-30 01:53:45.719993918 +0000 UTC m=+2149.298281253" Oct 30 01:53:45.802124 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:45.802097 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7492-bdc0f5" Oct 30 01:53:45.895367 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:45.895273 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88ed0eb1-f48a-4fe2-8bd3-9d2472ccdb2e-kube-api-access-hqdqc\" (UniqueName: \"kubernetes.io/projected/88ed0eb1-f48a-4fe2-8bd3-9d2472ccdb2e-kube-api-access-hqdqc\") pod \"88ed0eb1-f48a-4fe2-8bd3-9d2472ccdb2e\" (UID: \"88ed0eb1-f48a-4fe2-8bd3-9d2472ccdb2e\") " Oct 30 01:53:45.905061 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:45.905022 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88ed0eb1-f48a-4fe2-8bd3-9d2472ccdb2e-kube-api-access-hqdqc" pod "88ed0eb1-f48a-4fe2-8bd3-9d2472ccdb2e" (UID: "88ed0eb1-f48a-4fe2-8bd3-9d2472ccdb2e"). InnerVolumeSpecName "kube-api-access-hqdqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:53:45.997268 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:45.997236 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqdqc\" (UniqueName: \"kubernetes.io/projected/88ed0eb1-f48a-4fe2-8bd3-9d2472ccdb2e-kube-api-access-hqdqc\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:53:46.037234 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:46.037201 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" Oct 30 01:53:46.044295 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:46.044272 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" Oct 30 01:53:46.547906 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:46.547873 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88ed0eb1-f48a-4fe2-8bd3-9d2472ccdb2e" path="/var/lib/kubelet/pods/88ed0eb1-f48a-4fe2-8bd3-9d2472ccdb2e/volumes" Oct 30 01:53:46.677339 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:46.677310 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7492-bdc0f5" Oct 30 01:53:46.677583 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:46.677312 6221 scope.go:122] "RemoveContainer" containerID="1e2e9a0bb54eec807b121fc992d51464fef24ba9d16a761c48fb2c1a09599967" Oct 30 01:53:46.677583 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:46.677473 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" Oct 30 01:53:46.679133 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:46.679115 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" Oct 30 01:53:49.039603 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:49.039572 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-0835dd-7f7d4b44f9-zvnd7"] Oct 30 01:53:49.045342 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:49.045321 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-0835dd-7f7d4b44f9-zvnd7" Oct 30 01:53:49.066396 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:49.066362 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-0835dd-7f7d4b44f9-zvnd7"] Oct 30 01:53:49.074805 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:49.074777 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-0835dd-dockercfg-9xtmb\"" Oct 30 01:53:49.112152 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:49.112127 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-0835dd\"" Oct 30 01:53:49.160561 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:49.160443 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3b7a184-e07b-4bea-a3e1-6d2a5e939f09-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-0835dd-7f7d4b44f9-zvnd7\" (UID: \"b3b7a184-e07b-4bea-a3e1-6d2a5e939f09\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-0835dd-7f7d4b44f9-zvnd7" Oct 30 01:53:49.160767 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:49.160581 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsch5\" (UniqueName: \"kubernetes.io/projected/b3b7a184-e07b-4bea-a3e1-6d2a5e939f09-kube-api-access-qsch5\") pod \"payload-pre-processing-e2e-autores-0835dd-7f7d4b44f9-zvnd7\" (UID: \"b3b7a184-e07b-4bea-a3e1-6d2a5e939f09\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-0835dd-7f7d4b44f9-zvnd7" Oct 30 01:53:49.268564 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:49.267455 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3b7a184-e07b-4bea-a3e1-6d2a5e939f09-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-0835dd-7f7d4b44f9-zvnd7\" (UID: \"b3b7a184-e07b-4bea-a3e1-6d2a5e939f09\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-0835dd-7f7d4b44f9-zvnd7" Oct 30 01:53:49.268564 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:49.267602 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsch5\" (UniqueName: \"kubernetes.io/projected/b3b7a184-e07b-4bea-a3e1-6d2a5e939f09-kube-api-access-qsch5\") pod \"payload-pre-processing-e2e-autores-0835dd-7f7d4b44f9-zvnd7\" (UID: \"b3b7a184-e07b-4bea-a3e1-6d2a5e939f09\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-0835dd-7f7d4b44f9-zvnd7" Oct 30 01:53:49.271604 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:49.269050 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3b7a184-e07b-4bea-a3e1-6d2a5e939f09-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-0835dd-7f7d4b44f9-zvnd7\" (UID: \"b3b7a184-e07b-4bea-a3e1-6d2a5e939f09\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-0835dd-7f7d4b44f9-zvnd7" Oct 30 01:53:49.296055 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:49.295989 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsch5\" (UniqueName: \"kubernetes.io/projected/b3b7a184-e07b-4bea-a3e1-6d2a5e939f09-kube-api-access-qsch5\") pod \"payload-pre-processing-e2e-autores-0835dd-7f7d4b44f9-zvnd7\" (UID: \"b3b7a184-e07b-4bea-a3e1-6d2a5e939f09\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-0835dd-7f7d4b44f9-zvnd7" Oct 30 01:53:49.360488 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:49.360451 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-0835dd-7f7d4b44f9-zvnd7" Oct 30 01:53:49.694853 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:49.694826 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-0835dd-7f7d4b44f9-zvnd7"] Oct 30 01:53:49.697279 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:53:49.697252 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3b7a184_e07b_4bea_a3e1_6d2a5e939f09.slice/crio-64d1daef48e55c67dfa9aef8b922b74234ef84a6a403db4aba27576297c8362d WatchSource:0}: Error finding container 64d1daef48e55c67dfa9aef8b922b74234ef84a6a403db4aba27576297c8362d: Status 404 returned error can't find the container with id 64d1daef48e55c67dfa9aef8b922b74234ef84a6a403db4aba27576297c8362d Oct 30 01:53:50.708977 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:50.708942 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-0835dd-7f7d4b44f9-zvnd7" event={"ID":"b3b7a184-e07b-4bea-a3e1-6d2a5e939f09","Type":"ContainerStarted","Data":"a5a6b2492a27f4a9ade01275fac93f4f78364ff12bd705b94f231b586abf996d"} Oct 30 01:53:50.709605 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:50.708999 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-0835dd-7f7d4b44f9-zvnd7" event={"ID":"b3b7a184-e07b-4bea-a3e1-6d2a5e939f09","Type":"ContainerStarted","Data":"64d1daef48e55c67dfa9aef8b922b74234ef84a6a403db4aba27576297c8362d"} Oct 30 01:53:50.709605 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:50.709066 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-0835dd-7f7d4b44f9-zvnd7" Oct 30 01:53:51.317542 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:51.317471 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-0835dd-7f7d4b44f9-zvnd7" podStartSLOduration=3.3174547690000002 podStartE2EDuration="3.317454769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:53:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:53:50.76808078 +0000 UTC m=+2154.346368118" watchObservedRunningTime="2026-10-30 01:53:51.317454769 +0000 UTC m=+2154.895742098" Oct 30 01:53:51.319124 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:51.319103 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2eae39-d8f4c6966-szxzt"] Oct 30 01:53:51.323452 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:51.323435 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2eae39-d8f4c6966-szxzt" Oct 30 01:53:51.355198 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:51.350990 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-2eae39\"" Oct 30 01:53:51.355198 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:51.351035 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-2eae39-dockercfg-6br8x\"" Oct 30 01:53:51.355198 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:51.355109 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2eae39-d8f4c6966-szxzt"] Oct 30 01:53:51.520559 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:51.516841 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9vxm\" (UniqueName: \"kubernetes.io/projected/b683ef8c-fc50-4c69-98cf-fc3a1f59d6c1-kube-api-access-d9vxm\") pod \"payload-processing-e2e-extmod-a-2eae39-d8f4c6966-szxzt\" (UID: \"b683ef8c-fc50-4c69-98cf-fc3a1f59d6c1\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2eae39-d8f4c6966-szxzt" Oct 30 01:53:51.520559 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:51.517114 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b683ef8c-fc50-4c69-98cf-fc3a1f59d6c1-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-2eae39-d8f4c6966-szxzt\" (UID: \"b683ef8c-fc50-4c69-98cf-fc3a1f59d6c1\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2eae39-d8f4c6966-szxzt" Oct 30 01:53:51.568600 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:53:51.567989 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:53:51.624553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:51.620443 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9vxm\" (UniqueName: \"kubernetes.io/projected/b683ef8c-fc50-4c69-98cf-fc3a1f59d6c1-kube-api-access-d9vxm\") pod \"payload-processing-e2e-extmod-a-2eae39-d8f4c6966-szxzt\" (UID: \"b683ef8c-fc50-4c69-98cf-fc3a1f59d6c1\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2eae39-d8f4c6966-szxzt" Oct 30 01:53:51.624553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:51.620765 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b683ef8c-fc50-4c69-98cf-fc3a1f59d6c1-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-2eae39-d8f4c6966-szxzt\" (UID: \"b683ef8c-fc50-4c69-98cf-fc3a1f59d6c1\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2eae39-d8f4c6966-szxzt" Oct 30 01:53:51.626209 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:51.626160 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b683ef8c-fc50-4c69-98cf-fc3a1f59d6c1-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-2eae39-d8f4c6966-szxzt\" (UID: \"b683ef8c-fc50-4c69-98cf-fc3a1f59d6c1\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2eae39-d8f4c6966-szxzt" Oct 30 01:53:51.676545 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:51.675120 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9vxm\" (UniqueName: \"kubernetes.io/projected/b683ef8c-fc50-4c69-98cf-fc3a1f59d6c1-kube-api-access-d9vxm\") pod \"payload-processing-e2e-extmod-a-2eae39-d8f4c6966-szxzt\" (UID: \"b683ef8c-fc50-4c69-98cf-fc3a1f59d6c1\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-2eae39-d8f4c6966-szxzt" Oct 30 01:53:51.937060 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:51.936988 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2eae39-d8f4c6966-szxzt" Oct 30 01:53:52.237710 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:52.237685 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2eae39-d8f4c6966-szxzt"] Oct 30 01:53:52.240186 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:53:52.240147 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb683ef8c_fc50_4c69_98cf_fc3a1f59d6c1.slice/crio-83df210504d0299e34ccb40b9b1c82ce0b6fb7a6eca35564b43dfb8f431ef5f9 WatchSource:0}: Error finding container 83df210504d0299e34ccb40b9b1c82ce0b6fb7a6eca35564b43dfb8f431ef5f9: Status 404 returned error can't find the container with id 83df210504d0299e34ccb40b9b1c82ce0b6fb7a6eca35564b43dfb8f431ef5f9 Oct 30 01:53:52.734615 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:52.734576 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2eae39-d8f4c6966-szxzt" event={"ID":"b683ef8c-fc50-4c69-98cf-fc3a1f59d6c1","Type":"ContainerStarted","Data":"229e01cc8e36b137a5f9ff7d7d334862cbc6cfdd2db185e0d209f7cc71222c99"} Oct 30 01:53:52.734804 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:52.734657 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2eae39-d8f4c6966-szxzt" event={"ID":"b683ef8c-fc50-4c69-98cf-fc3a1f59d6c1","Type":"ContainerStarted","Data":"83df210504d0299e34ccb40b9b1c82ce0b6fb7a6eca35564b43dfb8f431ef5f9"} Oct 30 01:53:52.734804 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:52.734795 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-2eae39-d8f4c6966-szxzt" Oct 30 01:53:52.773523 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:53:52.773478 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-2eae39-d8f4c6966-szxzt" podStartSLOduration=1.7734673029999999 podStartE2EDuration="1.773467303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:53:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:53:52.772856671 +0000 UTC m=+2156.351144009" watchObservedRunningTime="2026-10-30 01:53:52.773467303 +0000 UTC m=+2156.351754638" Oct 30 01:54:01.719702 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:01.719668 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-0835dd-7f7d4b44f9-zvnd7" Oct 30 01:54:03.744405 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:03.744376 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-2eae39-d8f4c6966-szxzt" Oct 30 01:54:09.193608 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:09.192406 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-f1edaf-6947c677d8-fgcms"] Oct 30 01:54:09.270182 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:09.270150 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-f1edaf-6947c677d8-fgcms"] Oct 30 01:54:09.270496 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:09.270479 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-f1edaf-6947c677d8-fgcms" Oct 30 01:54:09.291122 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:09.291090 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-f1edaf-dockercfg-5nrcq\"" Oct 30 01:54:09.304760 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:09.304730 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-f1edaf\"" Oct 30 01:54:09.368201 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:09.368161 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll54v\" (UniqueName: \"kubernetes.io/projected/b124e8c4-1d49-4407-8841-99a368505a35-kube-api-access-ll54v\") pod \"payload-processing-e2e-extmod-b-f1edaf-6947c677d8-fgcms\" (UID: \"b124e8c4-1d49-4407-8841-99a368505a35\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-f1edaf-6947c677d8-fgcms" Oct 30 01:54:09.368375 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:09.368292 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b124e8c4-1d49-4407-8841-99a368505a35-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-f1edaf-6947c677d8-fgcms\" (UID: \"b124e8c4-1d49-4407-8841-99a368505a35\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-f1edaf-6947c677d8-fgcms" Oct 30 01:54:09.469590 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:09.469494 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll54v\" (UniqueName: \"kubernetes.io/projected/b124e8c4-1d49-4407-8841-99a368505a35-kube-api-access-ll54v\") pod \"payload-processing-e2e-extmod-b-f1edaf-6947c677d8-fgcms\" (UID: \"b124e8c4-1d49-4407-8841-99a368505a35\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-f1edaf-6947c677d8-fgcms" Oct 30 01:54:09.469737 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:09.469631 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b124e8c4-1d49-4407-8841-99a368505a35-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-f1edaf-6947c677d8-fgcms\" (UID: \"b124e8c4-1d49-4407-8841-99a368505a35\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-f1edaf-6947c677d8-fgcms" Oct 30 01:54:09.470364 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:09.470346 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b124e8c4-1d49-4407-8841-99a368505a35-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-f1edaf-6947c677d8-fgcms\" (UID: \"b124e8c4-1d49-4407-8841-99a368505a35\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-f1edaf-6947c677d8-fgcms" Oct 30 01:54:09.488503 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:09.488469 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll54v\" (UniqueName: \"kubernetes.io/projected/b124e8c4-1d49-4407-8841-99a368505a35-kube-api-access-ll54v\") pod \"payload-processing-e2e-extmod-b-f1edaf-6947c677d8-fgcms\" (UID: \"b124e8c4-1d49-4407-8841-99a368505a35\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-f1edaf-6947c677d8-fgcms" Oct 30 01:54:09.598550 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:09.598495 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-f1edaf-6947c677d8-fgcms" Oct 30 01:54:09.940402 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:09.940383 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-f1edaf-6947c677d8-fgcms"] Oct 30 01:54:09.943152 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:54:09.943125 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb124e8c4_1d49_4407_8841_99a368505a35.slice/crio-9c4abd7825472723cf55b7647041f573d310e64286b0e50f1035d41337e8e63b WatchSource:0}: Error finding container 9c4abd7825472723cf55b7647041f573d310e64286b0e50f1035d41337e8e63b: Status 404 returned error can't find the container with id 9c4abd7825472723cf55b7647041f573d310e64286b0e50f1035d41337e8e63b Oct 30 01:54:10.865898 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:10.865864 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-f1edaf-6947c677d8-fgcms" event={"ID":"b124e8c4-1d49-4407-8841-99a368505a35","Type":"ContainerStarted","Data":"6afe6533446cc041db65115a0f3d7d4bd187627015f3e4a82093426015466b2a"} Oct 30 01:54:10.866325 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:10.865916 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-f1edaf-6947c677d8-fgcms" event={"ID":"b124e8c4-1d49-4407-8841-99a368505a35","Type":"ContainerStarted","Data":"9c4abd7825472723cf55b7647041f573d310e64286b0e50f1035d41337e8e63b"} Oct 30 01:54:10.866325 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:10.866069 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-f1edaf-6947c677d8-fgcms" Oct 30 01:54:10.905565 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:10.905487 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-f1edaf-6947c677d8-fgcms" podStartSLOduration=1.9054688770000001 podStartE2EDuration="1.905468877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:54:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:54:10.903001644 +0000 UTC m=+2174.481288974" watchObservedRunningTime="2026-10-30 01:54:10.905468877 +0000 UTC m=+2174.483756214" Oct 30 01:54:17.580587 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:17.573460 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:54:21.495576 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:54:21.495518 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:54:21.875012 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:21.874985 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-f1edaf-6947c677d8-fgcms" Oct 30 01:54:22.816668 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:22.816638 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2eae39-58f6475475-r4tzv"] Oct 30 01:54:22.821150 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:22.821129 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2eae39-58f6475475-r4tzv" Oct 30 01:54:22.838437 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:22.838410 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2eae39-58f6475475-r4tzv"] Oct 30 01:54:22.849566 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:22.847872 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-2eae39-serving-cert\"" Oct 30 01:54:22.879681 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:22.879647 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e13139d7-1bcd-4eed-9614-afa7194cc175-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-2eae39-58f6475475-r4tzv\" (UID: \"e13139d7-1bcd-4eed-9614-afa7194cc175\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2eae39-58f6475475-r4tzv" Oct 30 01:54:22.879847 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:22.879794 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e13139d7-1bcd-4eed-9614-afa7194cc175-maas-api-tls\") pod \"maas-api-e2e-extmod-a-2eae39-58f6475475-r4tzv\" (UID: \"e13139d7-1bcd-4eed-9614-afa7194cc175\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2eae39-58f6475475-r4tzv" Oct 30 01:54:22.879947 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:22.879921 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e13139d7-1bcd-4eed-9614-afa7194cc175-metrics-certs\") pod \"maas-api-e2e-extmod-a-2eae39-58f6475475-r4tzv\" (UID: \"e13139d7-1bcd-4eed-9614-afa7194cc175\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2eae39-58f6475475-r4tzv" Oct 30 01:54:22.880115 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:22.880090 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p77gk\" (UniqueName: \"kubernetes.io/projected/e13139d7-1bcd-4eed-9614-afa7194cc175-kube-api-access-p77gk\") pod \"maas-api-e2e-extmod-a-2eae39-58f6475475-r4tzv\" (UID: \"e13139d7-1bcd-4eed-9614-afa7194cc175\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2eae39-58f6475475-r4tzv" Oct 30 01:54:22.981473 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:22.981416 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e13139d7-1bcd-4eed-9614-afa7194cc175-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-2eae39-58f6475475-r4tzv\" (UID: \"e13139d7-1bcd-4eed-9614-afa7194cc175\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2eae39-58f6475475-r4tzv" Oct 30 01:54:22.981704 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:22.981519 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e13139d7-1bcd-4eed-9614-afa7194cc175-maas-api-tls\") pod \"maas-api-e2e-extmod-a-2eae39-58f6475475-r4tzv\" (UID: \"e13139d7-1bcd-4eed-9614-afa7194cc175\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2eae39-58f6475475-r4tzv" Oct 30 01:54:22.981704 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:22.981585 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e13139d7-1bcd-4eed-9614-afa7194cc175-metrics-certs\") pod \"maas-api-e2e-extmod-a-2eae39-58f6475475-r4tzv\" (UID: \"e13139d7-1bcd-4eed-9614-afa7194cc175\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2eae39-58f6475475-r4tzv" Oct 30 01:54:22.981838 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:22.981705 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p77gk\" (UniqueName: \"kubernetes.io/projected/e13139d7-1bcd-4eed-9614-afa7194cc175-kube-api-access-p77gk\") pod \"maas-api-e2e-extmod-a-2eae39-58f6475475-r4tzv\" (UID: \"e13139d7-1bcd-4eed-9614-afa7194cc175\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2eae39-58f6475475-r4tzv" Oct 30 01:54:22.982173 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:22.982153 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e13139d7-1bcd-4eed-9614-afa7194cc175-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-2eae39-58f6475475-r4tzv\" (UID: \"e13139d7-1bcd-4eed-9614-afa7194cc175\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2eae39-58f6475475-r4tzv" Oct 30 01:54:22.983993 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:22.983972 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e13139d7-1bcd-4eed-9614-afa7194cc175-maas-api-tls\") pod \"maas-api-e2e-extmod-a-2eae39-58f6475475-r4tzv\" (UID: \"e13139d7-1bcd-4eed-9614-afa7194cc175\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2eae39-58f6475475-r4tzv" Oct 30 01:54:22.984080 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:22.984012 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e13139d7-1bcd-4eed-9614-afa7194cc175-metrics-certs\") pod \"maas-api-e2e-extmod-a-2eae39-58f6475475-r4tzv\" (UID: \"e13139d7-1bcd-4eed-9614-afa7194cc175\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2eae39-58f6475475-r4tzv" Oct 30 01:54:23.001736 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:23.001709 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p77gk\" (UniqueName: \"kubernetes.io/projected/e13139d7-1bcd-4eed-9614-afa7194cc175-kube-api-access-p77gk\") pod \"maas-api-e2e-extmod-a-2eae39-58f6475475-r4tzv\" (UID: \"e13139d7-1bcd-4eed-9614-afa7194cc175\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2eae39-58f6475475-r4tzv" Oct 30 01:54:23.138916 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:23.138826 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2eae39-58f6475475-r4tzv" Oct 30 01:54:23.445722 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:23.445694 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2eae39-58f6475475-r4tzv"] Oct 30 01:54:23.448765 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:54:23.448738 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13139d7_1bcd_4eed_9614_afa7194cc175.slice/crio-aca52c833f511ace106eb432d1773b4af5af2a1a6cf374f58ab64fc0c3687c0e WatchSource:0}: Error finding container aca52c833f511ace106eb432d1773b4af5af2a1a6cf374f58ab64fc0c3687c0e: Status 404 returned error can't find the container with id aca52c833f511ace106eb432d1773b4af5af2a1a6cf374f58ab64fc0c3687c0e Oct 30 01:54:23.450292 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:23.450274 6221 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 01:54:23.969461 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:23.969411 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2eae39-58f6475475-r4tzv" event={"ID":"e13139d7-1bcd-4eed-9614-afa7194cc175","Type":"ContainerStarted","Data":"b869c5e6f02ec6dc07f6c5c513742b2526aa1c73c4106918ec78694d37db23cd"} Oct 30 01:54:23.969927 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:23.969488 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2eae39-58f6475475-r4tzv" event={"ID":"e13139d7-1bcd-4eed-9614-afa7194cc175","Type":"ContainerStarted","Data":"aca52c833f511ace106eb432d1773b4af5af2a1a6cf374f58ab64fc0c3687c0e"} Oct 30 01:54:23.969927 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:23.969650 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2eae39-58f6475475-r4tzv" Oct 30 01:54:29.982212 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:29.982181 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2eae39-58f6475475-r4tzv" Oct 30 01:54:30.028705 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:30.028648 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2eae39-58f6475475-r4tzv" podStartSLOduration=7.672794811 podStartE2EDuration="8.028633367s" totalImagesPullingTime="355.838556ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:54:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:54:24.082125282 +0000 UTC m=+2187.660412599" watchObservedRunningTime="2026-10-30 01:54:30.028633367 +0000 UTC m=+2193.606920703" Oct 30 01:54:51.495025 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:54:51.494987 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:54:57.237302 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:57.237215 6221 generic.go:356] "Generic (PLEG): container finished" podID="dcd89198-b4a1-40cd-9850-3dda3af7bbab" containerID="aa3a19507a0829c2b275082658f45f955b9aa27e73a1190ae70ece556cb20d8c" exitCode=0 Oct 30 01:54:57.237302 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:57.237287 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" event={"ID":"dcd89198-b4a1-40cd-9850-3dda3af7bbab","Type":"ContainerDied","Data":"aa3a19507a0829c2b275082658f45f955b9aa27e73a1190ae70ece556cb20d8c"} Oct 30 01:54:57.237880 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:57.237754 6221 scope.go:122] "RemoveContainer" containerID="aa3a19507a0829c2b275082658f45f955b9aa27e73a1190ae70ece556cb20d8c" Oct 30 01:54:57.239524 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:57.239497 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:54:57.239616 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:57.239582 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:54:57.239655 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:57.239628 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:54:58.261327 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:58.261021 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" event={"ID":"dcd89198-b4a1-40cd-9850-3dda3af7bbab","Type":"ContainerStarted","Data":"c81b77470d2ba487be1a1010d6da2c84db607c61e7ca0f6fdcd4f557cb1f022a"} Oct 30 01:54:59.037281 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:59.037244 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" Oct 30 01:54:59.038700 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:59.038669 6221 patch_prober.go:29] interesting pod/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 01:54:59.038855 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:54:59.038738 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" podUID="dcd89198-b4a1-40cd-9850-3dda3af7bbab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 01:55:00.037559 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:00.037487 6221 patch_prober.go:29] interesting pod/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 01:55:00.038112 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:00.037589 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" podUID="dcd89198-b4a1-40cd-9850-3dda3af7bbab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 01:55:01.037046 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:01.037009 6221 patch_prober.go:29] interesting pod/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 01:55:01.037235 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:01.037074 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" podUID="dcd89198-b4a1-40cd-9850-3dda3af7bbab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 01:55:01.677987 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:01.677947 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" Oct 30 01:55:02.036886 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:02.036843 6221 patch_prober.go:29] interesting pod/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 01:55:02.037085 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:02.036912 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" podUID="dcd89198-b4a1-40cd-9850-3dda3af7bbab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 01:55:03.036970 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:03.036930 6221 patch_prober.go:29] interesting pod/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 01:55:03.037617 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:03.037583 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" podUID="dcd89198-b4a1-40cd-9850-3dda3af7bbab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 01:55:04.037369 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:04.037317 6221 patch_prober.go:29] interesting pod/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" start-of-body= Oct 30 01:55:04.037926 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:04.037390 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" podUID="dcd89198-b4a1-40cd-9850-3dda3af7bbab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.169:15021/healthz/ready\": dial tcp 10.134.0.169:15021: connect: connection refused" Oct 30 01:55:05.042511 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:05.042468 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" Oct 30 01:55:05.043989 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:05.043962 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" Oct 30 01:55:05.126408 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:05.126376 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-093d23-5fd64fb8d4-z266p"] Oct 30 01:55:05.129980 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:05.129956 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-093d23-5fd64fb8d4-z266p" Oct 30 01:55:05.151240 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:05.151206 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-093d23\"" Oct 30 01:55:05.151403 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:05.151332 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-093d23-dockercfg-hn79p\"" Oct 30 01:55:05.154483 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:05.153900 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-093d23-5fd64fb8d4-z266p"] Oct 30 01:55:05.204236 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:05.204178 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab3bacd1-4d0c-4c6b-b672-582c8d729702-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-093d23-5fd64fb8d4-z266p\" (UID: \"ab3bacd1-4d0c-4c6b-b672-582c8d729702\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-093d23-5fd64fb8d4-z266p" Oct 30 01:55:05.204389 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:05.204346 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz92k\" (UniqueName: \"kubernetes.io/projected/ab3bacd1-4d0c-4c6b-b672-582c8d729702-kube-api-access-hz92k\") pod \"payload-processing-e2e-extmod-b-093d23-5fd64fb8d4-z266p\" (UID: \"ab3bacd1-4d0c-4c6b-b672-582c8d729702\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-093d23-5fd64fb8d4-z266p" Oct 30 01:55:05.305560 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:05.305448 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab3bacd1-4d0c-4c6b-b672-582c8d729702-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-093d23-5fd64fb8d4-z266p\" (UID: \"ab3bacd1-4d0c-4c6b-b672-582c8d729702\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-093d23-5fd64fb8d4-z266p" Oct 30 01:55:05.305728 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:05.305576 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz92k\" (UniqueName: \"kubernetes.io/projected/ab3bacd1-4d0c-4c6b-b672-582c8d729702-kube-api-access-hz92k\") pod \"payload-processing-e2e-extmod-b-093d23-5fd64fb8d4-z266p\" (UID: \"ab3bacd1-4d0c-4c6b-b672-582c8d729702\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-093d23-5fd64fb8d4-z266p" Oct 30 01:55:05.306273 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:05.306254 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab3bacd1-4d0c-4c6b-b672-582c8d729702-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-093d23-5fd64fb8d4-z266p\" (UID: \"ab3bacd1-4d0c-4c6b-b672-582c8d729702\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-093d23-5fd64fb8d4-z266p" Oct 30 01:55:05.326485 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:05.326439 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz92k\" (UniqueName: \"kubernetes.io/projected/ab3bacd1-4d0c-4c6b-b672-582c8d729702-kube-api-access-hz92k\") pod \"payload-processing-e2e-extmod-b-093d23-5fd64fb8d4-z266p\" (UID: \"ab3bacd1-4d0c-4c6b-b672-582c8d729702\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-093d23-5fd64fb8d4-z266p" Oct 30 01:55:05.449555 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:05.448824 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-093d23-5fd64fb8d4-z266p" Oct 30 01:55:05.868573 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:05.868545 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-093d23-5fd64fb8d4-z266p"] Oct 30 01:55:05.870908 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:55:05.870839 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab3bacd1_4d0c_4c6b_b672_582c8d729702.slice/crio-6ad6c1bba46d3aaf6fed6cfad22572898db799069ad4c4d3c7f5aed2f73e32f7 WatchSource:0}: Error finding container 6ad6c1bba46d3aaf6fed6cfad22572898db799069ad4c4d3c7f5aed2f73e32f7: Status 404 returned error can't find the container with id 6ad6c1bba46d3aaf6fed6cfad22572898db799069ad4c4d3c7f5aed2f73e32f7 Oct 30 01:55:06.328337 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:06.328296 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-093d23-5fd64fb8d4-z266p" event={"ID":"ab3bacd1-4d0c-4c6b-b672-582c8d729702","Type":"ContainerStarted","Data":"8cd2a22f2763c98a2c36ba8b6919a437e82cc0a632e323737e7b0e3b5d705cb7"} Oct 30 01:55:06.328862 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:06.328361 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-093d23-5fd64fb8d4-z266p" Oct 30 01:55:06.328862 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:06.328385 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-093d23-5fd64fb8d4-z266p" event={"ID":"ab3bacd1-4d0c-4c6b-b672-582c8d729702","Type":"ContainerStarted","Data":"6ad6c1bba46d3aaf6fed6cfad22572898db799069ad4c4d3c7f5aed2f73e32f7"} Oct 30 01:55:06.372292 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:06.372240 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-093d23-5fd64fb8d4-z266p" podStartSLOduration=1.372226433 podStartE2EDuration="1.372226433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:55:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:55:06.371629247 +0000 UTC m=+2229.949916585" watchObservedRunningTime="2026-10-30 01:55:06.372226433 +0000 UTC m=+2229.950513759" Oct 30 01:55:06.466963 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:06.466924 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-0835dd/autores-model-0835dd-kserve-7b59855b5-5kjp6"] Oct 30 01:55:06.471667 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:06.471632 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-0835dd/autores-model-0835dd-kserve-7b59855b5-5kjp6" Oct 30 01:55:06.493621 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:06.493586 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-0835dd/autores-model-0835dd-kserve-7b59855b5-5kjp6"] Oct 30 01:55:06.495913 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:06.495879 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-0835dd\"/\"kube-root-ca.crt\"" Oct 30 01:55:06.496179 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:06.496138 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-0835dd\"/\"openshift-service-ca.crt\"" Oct 30 01:55:06.496179 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:06.496170 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-0835dd\"/\"default-dockercfg-b5v26\"" Oct 30 01:55:06.496356 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:06.496263 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-0835dd\"/\"autores-model-0835dd-kserve-self-signed-certs\"" Oct 30 01:55:06.515357 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:06.515321 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:55:06.533782 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:06.528660 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55b4ef19-8d44-4d6b-95d2-d17c41cd49d6-dshm\") pod \"autores-model-0835dd-kserve-7b59855b5-5kjp6\" (UID: \"55b4ef19-8d44-4d6b-95d2-d17c41cd49d6\") " pod="ai-tenant-e2e-autores-0835dd/autores-model-0835dd-kserve-7b59855b5-5kjp6" Oct 30 01:55:06.533782 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:06.528797 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55b4ef19-8d44-4d6b-95d2-d17c41cd49d6-home\") pod \"autores-model-0835dd-kserve-7b59855b5-5kjp6\" (UID: \"55b4ef19-8d44-4d6b-95d2-d17c41cd49d6\") " pod="ai-tenant-e2e-autores-0835dd/autores-model-0835dd-kserve-7b59855b5-5kjp6" Oct 30 01:55:06.533782 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:06.528866 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55b4ef19-8d44-4d6b-95d2-d17c41cd49d6-model-cache\") pod \"autores-model-0835dd-kserve-7b59855b5-5kjp6\" (UID: \"55b4ef19-8d44-4d6b-95d2-d17c41cd49d6\") " pod="ai-tenant-e2e-autores-0835dd/autores-model-0835dd-kserve-7b59855b5-5kjp6" Oct 30 01:55:06.533782 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:06.528891 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55b4ef19-8d44-4d6b-95d2-d17c41cd49d6-tls-certs\") pod \"autores-model-0835dd-kserve-7b59855b5-5kjp6\" (UID: \"55b4ef19-8d44-4d6b-95d2-d17c41cd49d6\") " pod="ai-tenant-e2e-autores-0835dd/autores-model-0835dd-kserve-7b59855b5-5kjp6" Oct 30 01:55:06.533782 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:06.529012 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55b4ef19-8d44-4d6b-95d2-d17c41cd49d6-tmp-dir\") pod \"autores-model-0835dd-kserve-7b59855b5-5kjp6\" (UID: \"55b4ef19-8d44-4d6b-95d2-d17c41cd49d6\") " pod="ai-tenant-e2e-autores-0835dd/autores-model-0835dd-kserve-7b59855b5-5kjp6" Oct 30 01:55:06.533782 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:06.529047 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bvl8\" (UniqueName: \"kubernetes.io/projected/55b4ef19-8d44-4d6b-95d2-d17c41cd49d6-kube-api-access-7bvl8\") pod \"autores-model-0835dd-kserve-7b59855b5-5kjp6\" (UID: \"55b4ef19-8d44-4d6b-95d2-d17c41cd49d6\") " pod="ai-tenant-e2e-autores-0835dd/autores-model-0835dd-kserve-7b59855b5-5kjp6" Oct 30 01:55:06.630850 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:06.630764 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bvl8\" (UniqueName: \"kubernetes.io/projected/55b4ef19-8d44-4d6b-95d2-d17c41cd49d6-kube-api-access-7bvl8\") pod \"autores-model-0835dd-kserve-7b59855b5-5kjp6\" (UID: \"55b4ef19-8d44-4d6b-95d2-d17c41cd49d6\") " pod="ai-tenant-e2e-autores-0835dd/autores-model-0835dd-kserve-7b59855b5-5kjp6" Oct 30 01:55:06.631017 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:06.630890 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55b4ef19-8d44-4d6b-95d2-d17c41cd49d6-dshm\") pod \"autores-model-0835dd-kserve-7b59855b5-5kjp6\" (UID: \"55b4ef19-8d44-4d6b-95d2-d17c41cd49d6\") " pod="ai-tenant-e2e-autores-0835dd/autores-model-0835dd-kserve-7b59855b5-5kjp6" Oct 30 01:55:06.631017 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:06.630973 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55b4ef19-8d44-4d6b-95d2-d17c41cd49d6-home\") pod \"autores-model-0835dd-kserve-7b59855b5-5kjp6\" (UID: \"55b4ef19-8d44-4d6b-95d2-d17c41cd49d6\") " pod="ai-tenant-e2e-autores-0835dd/autores-model-0835dd-kserve-7b59855b5-5kjp6" Oct 30 01:55:06.631121 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:06.631025 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55b4ef19-8d44-4d6b-95d2-d17c41cd49d6-model-cache\") pod \"autores-model-0835dd-kserve-7b59855b5-5kjp6\" (UID: \"55b4ef19-8d44-4d6b-95d2-d17c41cd49d6\") " pod="ai-tenant-e2e-autores-0835dd/autores-model-0835dd-kserve-7b59855b5-5kjp6" Oct 30 01:55:06.631121 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:06.631048 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55b4ef19-8d44-4d6b-95d2-d17c41cd49d6-tls-certs\") pod \"autores-model-0835dd-kserve-7b59855b5-5kjp6\" (UID: \"55b4ef19-8d44-4d6b-95d2-d17c41cd49d6\") " pod="ai-tenant-e2e-autores-0835dd/autores-model-0835dd-kserve-7b59855b5-5kjp6" Oct 30 01:55:06.631332 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:06.631297 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55b4ef19-8d44-4d6b-95d2-d17c41cd49d6-tmp-dir\") pod \"autores-model-0835dd-kserve-7b59855b5-5kjp6\" (UID: \"55b4ef19-8d44-4d6b-95d2-d17c41cd49d6\") " pod="ai-tenant-e2e-autores-0835dd/autores-model-0835dd-kserve-7b59855b5-5kjp6" Oct 30 01:55:06.631506 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:06.631426 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55b4ef19-8d44-4d6b-95d2-d17c41cd49d6-home\") pod \"autores-model-0835dd-kserve-7b59855b5-5kjp6\" (UID: \"55b4ef19-8d44-4d6b-95d2-d17c41cd49d6\") " pod="ai-tenant-e2e-autores-0835dd/autores-model-0835dd-kserve-7b59855b5-5kjp6" Oct 30 01:55:06.631506 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:06.631492 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55b4ef19-8d44-4d6b-95d2-d17c41cd49d6-model-cache\") pod \"autores-model-0835dd-kserve-7b59855b5-5kjp6\" (UID: \"55b4ef19-8d44-4d6b-95d2-d17c41cd49d6\") " pod="ai-tenant-e2e-autores-0835dd/autores-model-0835dd-kserve-7b59855b5-5kjp6" Oct 30 01:55:06.631659 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:06.631609 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55b4ef19-8d44-4d6b-95d2-d17c41cd49d6-tmp-dir\") pod \"autores-model-0835dd-kserve-7b59855b5-5kjp6\" (UID: \"55b4ef19-8d44-4d6b-95d2-d17c41cd49d6\") " pod="ai-tenant-e2e-autores-0835dd/autores-model-0835dd-kserve-7b59855b5-5kjp6" Oct 30 01:55:06.633303 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:06.633281 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55b4ef19-8d44-4d6b-95d2-d17c41cd49d6-dshm\") pod \"autores-model-0835dd-kserve-7b59855b5-5kjp6\" (UID: \"55b4ef19-8d44-4d6b-95d2-d17c41cd49d6\") " pod="ai-tenant-e2e-autores-0835dd/autores-model-0835dd-kserve-7b59855b5-5kjp6" Oct 30 01:55:06.633608 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:06.633589 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55b4ef19-8d44-4d6b-95d2-d17c41cd49d6-tls-certs\") pod \"autores-model-0835dd-kserve-7b59855b5-5kjp6\" (UID: \"55b4ef19-8d44-4d6b-95d2-d17c41cd49d6\") " pod="ai-tenant-e2e-autores-0835dd/autores-model-0835dd-kserve-7b59855b5-5kjp6" Oct 30 01:55:06.653363 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:06.653327 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bvl8\" (UniqueName: \"kubernetes.io/projected/55b4ef19-8d44-4d6b-95d2-d17c41cd49d6-kube-api-access-7bvl8\") pod \"autores-model-0835dd-kserve-7b59855b5-5kjp6\" (UID: \"55b4ef19-8d44-4d6b-95d2-d17c41cd49d6\") " pod="ai-tenant-e2e-autores-0835dd/autores-model-0835dd-kserve-7b59855b5-5kjp6" Oct 30 01:55:06.786835 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:06.786796 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-0835dd/autores-model-0835dd-kserve-7b59855b5-5kjp6" Oct 30 01:55:07.307278 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:07.293658 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-0835dd/autores-model-0835dd-kserve-7b59855b5-5kjp6"] Oct 30 01:55:07.372214 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:07.372181 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-0835dd/autores-model-0835dd-kserve-7b59855b5-5kjp6" event={"ID":"55b4ef19-8d44-4d6b-95d2-d17c41cd49d6","Type":"ContainerStarted","Data":"5140e0e45592abd66514514d029413ab0a8a11f6e55aa5e2aa68716e183d51c0"} Oct 30 01:55:08.380479 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:08.380432 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-0835dd/autores-model-0835dd-kserve-7b59855b5-5kjp6" event={"ID":"55b4ef19-8d44-4d6b-95d2-d17c41cd49d6","Type":"ContainerStarted","Data":"b9bb827914d2fa1e83daf5178194363df8402693787f16f7f40a8ee3b626ed6f"} Oct 30 01:55:08.424108 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:08.424051 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-0835dd/autores-model-0835dd-kserve-7b59855b5-5kjp6" podStartSLOduration=2.424036788 podStartE2EDuration="2.424036788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:55:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:55:08.420243039 +0000 UTC m=+2231.998530375" watchObservedRunningTime="2026-10-30 01:55:08.424036788 +0000 UTC m=+2232.002324124" Oct 30 01:55:08.787186 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:08.787141 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-0835dd/autores-model-0835dd-kserve-7b59855b5-5kjp6" Oct 30 01:55:12.775980 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:12.775944 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-cd8173-5674dfcc9f-sqswj"] Oct 30 01:55:12.779316 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:12.779298 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cd8173-5674dfcc9f-sqswj" Oct 30 01:55:12.800566 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:12.800525 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-cd8173-5674dfcc9f-sqswj"] Oct 30 01:55:12.809045 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:12.808998 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-cd8173\"" Oct 30 01:55:12.809217 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:12.809124 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-cd8173-dockercfg-n8js8\"" Oct 30 01:55:12.862578 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:12.852827 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a08e7d0c-aaf7-4960-bc25-23e2290c1bd6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-cd8173-5674dfcc9f-sqswj\" (UID: \"a08e7d0c-aaf7-4960-bc25-23e2290c1bd6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cd8173-5674dfcc9f-sqswj" Oct 30 01:55:12.863023 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:12.862985 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnl8c\" (UniqueName: \"kubernetes.io/projected/a08e7d0c-aaf7-4960-bc25-23e2290c1bd6-kube-api-access-jnl8c\") pod \"payload-pre-processing-e2e-extmod-b-cd8173-5674dfcc9f-sqswj\" (UID: \"a08e7d0c-aaf7-4960-bc25-23e2290c1bd6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cd8173-5674dfcc9f-sqswj" Oct 30 01:55:12.978551 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:12.977425 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a08e7d0c-aaf7-4960-bc25-23e2290c1bd6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-cd8173-5674dfcc9f-sqswj\" (UID: \"a08e7d0c-aaf7-4960-bc25-23e2290c1bd6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cd8173-5674dfcc9f-sqswj" Oct 30 01:55:12.978551 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:12.977511 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnl8c\" (UniqueName: \"kubernetes.io/projected/a08e7d0c-aaf7-4960-bc25-23e2290c1bd6-kube-api-access-jnl8c\") pod \"payload-pre-processing-e2e-extmod-b-cd8173-5674dfcc9f-sqswj\" (UID: \"a08e7d0c-aaf7-4960-bc25-23e2290c1bd6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cd8173-5674dfcc9f-sqswj" Oct 30 01:55:12.999554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:12.994088 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a08e7d0c-aaf7-4960-bc25-23e2290c1bd6-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-cd8173-5674dfcc9f-sqswj\" (UID: \"a08e7d0c-aaf7-4960-bc25-23e2290c1bd6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cd8173-5674dfcc9f-sqswj" Oct 30 01:55:13.030625 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:13.025739 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnl8c\" (UniqueName: \"kubernetes.io/projected/a08e7d0c-aaf7-4960-bc25-23e2290c1bd6-kube-api-access-jnl8c\") pod \"payload-pre-processing-e2e-extmod-b-cd8173-5674dfcc9f-sqswj\" (UID: \"a08e7d0c-aaf7-4960-bc25-23e2290c1bd6\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cd8173-5674dfcc9f-sqswj" Oct 30 01:55:13.092420 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:13.092382 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cd8173-5674dfcc9f-sqswj" Oct 30 01:55:13.395059 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:13.395031 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-cd8173-5674dfcc9f-sqswj"] Oct 30 01:55:13.397768 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:55:13.397739 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda08e7d0c_aaf7_4960_bc25_23e2290c1bd6.slice/crio-961688c58d3ecd1d3fd3e84602010533bfd5dee24e589c394c7708b5db444e5b WatchSource:0}: Error finding container 961688c58d3ecd1d3fd3e84602010533bfd5dee24e589c394c7708b5db444e5b: Status 404 returned error can't find the container with id 961688c58d3ecd1d3fd3e84602010533bfd5dee24e589c394c7708b5db444e5b Oct 30 01:55:13.415850 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:13.415811 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cd8173-5674dfcc9f-sqswj" event={"ID":"a08e7d0c-aaf7-4960-bc25-23e2290c1bd6","Type":"ContainerStarted","Data":"961688c58d3ecd1d3fd3e84602010533bfd5dee24e589c394c7708b5db444e5b"} Oct 30 01:55:14.425258 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:14.425224 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cd8173-5674dfcc9f-sqswj" event={"ID":"a08e7d0c-aaf7-4960-bc25-23e2290c1bd6","Type":"ContainerStarted","Data":"b9fbebb8f49d5e6c57a95b7244372e0a697cd501df678fe2ff23771462cdb5ef"} Oct 30 01:55:14.425701 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:14.425296 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cd8173-5674dfcc9f-sqswj" Oct 30 01:55:14.465949 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:14.465899 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cd8173-5674dfcc9f-sqswj" podStartSLOduration=2.465883467 podStartE2EDuration="2.465883467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:55:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:55:14.46411739 +0000 UTC m=+2238.042404727" watchObservedRunningTime="2026-10-30 01:55:14.465883467 +0000 UTC m=+2238.044170805" Oct 30 01:55:16.787731 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:16.787695 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-0835dd/autores-model-0835dd-kserve-7b59855b5-5kjp6" Oct 30 01:55:16.800262 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:16.800233 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-0835dd/autores-model-0835dd-kserve-7b59855b5-5kjp6" Oct 30 01:55:17.376401 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:17.376372 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-093d23-5fd64fb8d4-z266p" Oct 30 01:55:17.464764 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:17.464737 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-0835dd/autores-model-0835dd-kserve-7b59855b5-5kjp6" Oct 30 01:55:21.498692 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:55:21.498649 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:55:25.435345 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:25.435314 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cd8173-5674dfcc9f-sqswj" Oct 30 01:55:31.888796 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:31.888758 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-0835dd/autores-model-0835dd-kserve-7b59855b5-5kjp6"] Oct 30 01:55:37.170746 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:37.170710 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj"] Oct 30 01:55:37.171184 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:37.170905 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj" podUID="3e8eba77-efd7-4b5c-8093-4af09c7dce5c" containerName="istio-proxy" containerID="cri-o://cc04c43519df2931e1e23a15cc336d7e5f5780a01a26544100645dc3f197d73e" gracePeriod=30 Oct 30 01:55:40.069081 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:40.068996 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-093d23-55b74987dd-8nwrk"] Oct 30 01:55:40.084452 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:40.084420 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-093d23-55b74987dd-8nwrk" Oct 30 01:55:40.097908 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:40.097882 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-093d23-55b74987dd-8nwrk"] Oct 30 01:55:40.106356 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:40.106328 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-093d23-serving-cert\"" Oct 30 01:55:40.160670 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:40.160633 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08f6b864-02b6-4aa9-8616-777fb088d0a4-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-093d23-55b74987dd-8nwrk\" (UID: \"08f6b864-02b6-4aa9-8616-777fb088d0a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-093d23-55b74987dd-8nwrk" Oct 30 01:55:40.160842 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:40.160798 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08f6b864-02b6-4aa9-8616-777fb088d0a4-metrics-certs\") pod \"maas-api-e2e-extmod-b-093d23-55b74987dd-8nwrk\" (UID: \"08f6b864-02b6-4aa9-8616-777fb088d0a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-093d23-55b74987dd-8nwrk" Oct 30 01:55:40.160885 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:40.160861 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77g8j\" (UniqueName: \"kubernetes.io/projected/08f6b864-02b6-4aa9-8616-777fb088d0a4-kube-api-access-77g8j\") pod \"maas-api-e2e-extmod-b-093d23-55b74987dd-8nwrk\" (UID: \"08f6b864-02b6-4aa9-8616-777fb088d0a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-093d23-55b74987dd-8nwrk" Oct 30 01:55:40.160955 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:40.160939 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08f6b864-02b6-4aa9-8616-777fb088d0a4-maas-api-tls\") pod \"maas-api-e2e-extmod-b-093d23-55b74987dd-8nwrk\" (UID: \"08f6b864-02b6-4aa9-8616-777fb088d0a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-093d23-55b74987dd-8nwrk" Oct 30 01:55:40.262887 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:40.262847 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08f6b864-02b6-4aa9-8616-777fb088d0a4-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-093d23-55b74987dd-8nwrk\" (UID: \"08f6b864-02b6-4aa9-8616-777fb088d0a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-093d23-55b74987dd-8nwrk" Oct 30 01:55:40.263096 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:40.263005 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08f6b864-02b6-4aa9-8616-777fb088d0a4-metrics-certs\") pod \"maas-api-e2e-extmod-b-093d23-55b74987dd-8nwrk\" (UID: \"08f6b864-02b6-4aa9-8616-777fb088d0a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-093d23-55b74987dd-8nwrk" Oct 30 01:55:40.263096 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:40.263084 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77g8j\" (UniqueName: \"kubernetes.io/projected/08f6b864-02b6-4aa9-8616-777fb088d0a4-kube-api-access-77g8j\") pod \"maas-api-e2e-extmod-b-093d23-55b74987dd-8nwrk\" (UID: \"08f6b864-02b6-4aa9-8616-777fb088d0a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-093d23-55b74987dd-8nwrk" Oct 30 01:55:40.263217 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:40.263157 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08f6b864-02b6-4aa9-8616-777fb088d0a4-maas-api-tls\") pod \"maas-api-e2e-extmod-b-093d23-55b74987dd-8nwrk\" (UID: \"08f6b864-02b6-4aa9-8616-777fb088d0a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-093d23-55b74987dd-8nwrk" Oct 30 01:55:40.263592 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:40.263568 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08f6b864-02b6-4aa9-8616-777fb088d0a4-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-093d23-55b74987dd-8nwrk\" (UID: \"08f6b864-02b6-4aa9-8616-777fb088d0a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-093d23-55b74987dd-8nwrk" Oct 30 01:55:40.265393 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:40.265369 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08f6b864-02b6-4aa9-8616-777fb088d0a4-metrics-certs\") pod \"maas-api-e2e-extmod-b-093d23-55b74987dd-8nwrk\" (UID: \"08f6b864-02b6-4aa9-8616-777fb088d0a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-093d23-55b74987dd-8nwrk" Oct 30 01:55:40.265585 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:40.265559 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08f6b864-02b6-4aa9-8616-777fb088d0a4-maas-api-tls\") pod \"maas-api-e2e-extmod-b-093d23-55b74987dd-8nwrk\" (UID: \"08f6b864-02b6-4aa9-8616-777fb088d0a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-093d23-55b74987dd-8nwrk" Oct 30 01:55:40.284250 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:40.284220 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77g8j\" (UniqueName: \"kubernetes.io/projected/08f6b864-02b6-4aa9-8616-777fb088d0a4-kube-api-access-77g8j\") pod \"maas-api-e2e-extmod-b-093d23-55b74987dd-8nwrk\" (UID: \"08f6b864-02b6-4aa9-8616-777fb088d0a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-093d23-55b74987dd-8nwrk" Oct 30 01:55:40.400642 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:40.399854 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-093d23-55b74987dd-8nwrk" Oct 30 01:55:40.716700 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:40.716668 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-093d23-55b74987dd-8nwrk"] Oct 30 01:55:40.719688 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:55:40.719657 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08f6b864_02b6_4aa9_8616_777fb088d0a4.slice/crio-75fc19b6c13106a24e238550939c77e36171da17385af5fa140c0958a41ea828 WatchSource:0}: Error finding container 75fc19b6c13106a24e238550939c77e36171da17385af5fa140c0958a41ea828: Status 404 returned error can't find the container with id 75fc19b6c13106a24e238550939c77e36171da17385af5fa140c0958a41ea828 Oct 30 01:55:41.670326 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:41.670287 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-093d23-55b74987dd-8nwrk" event={"ID":"08f6b864-02b6-4aa9-8616-777fb088d0a4","Type":"ContainerStarted","Data":"9712d9e65affa092ac889d7137c0d6cdd73586382cb5fd9323f9d67f4e796c93"} Oct 30 01:55:41.670881 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:41.670351 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-093d23-55b74987dd-8nwrk" Oct 30 01:55:41.670881 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:41.670369 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-093d23-55b74987dd-8nwrk" event={"ID":"08f6b864-02b6-4aa9-8616-777fb088d0a4","Type":"ContainerStarted","Data":"75fc19b6c13106a24e238550939c77e36171da17385af5fa140c0958a41ea828"} Oct 30 01:55:41.713185 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:41.713114 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-093d23-55b74987dd-8nwrk" podStartSLOduration=1.389949356 podStartE2EDuration="1.713099665s" totalImagesPullingTime="323.150309ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:55:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:55:41.712594916 +0000 UTC m=+2265.290882252" watchObservedRunningTime="2026-10-30 01:55:41.713099665 +0000 UTC m=+2265.291387001" Oct 30 01:55:42.449571 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:42.448628 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj" Oct 30 01:55:42.612914 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:42.612471 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-workload-socket\") pod \"3e8eba77-efd7-4b5c-8093-4af09c7dce5c\" (UID: \"3e8eba77-efd7-4b5c-8093-4af09c7dce5c\") " Oct 30 01:55:42.616553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:42.616306 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-kube-api-access-k4tsv\" (UniqueName: \"kubernetes.io/projected/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-kube-api-access-k4tsv\") pod \"3e8eba77-efd7-4b5c-8093-4af09c7dce5c\" (UID: \"3e8eba77-efd7-4b5c-8093-4af09c7dce5c\") " Oct 30 01:55:42.616553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:42.616393 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-istio-podinfo\") pod \"3e8eba77-efd7-4b5c-8093-4af09c7dce5c\" (UID: \"3e8eba77-efd7-4b5c-8093-4af09c7dce5c\") " Oct 30 01:55:42.616553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:42.616494 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-istio-envoy\") pod \"3e8eba77-efd7-4b5c-8093-4af09c7dce5c\" (UID: \"3e8eba77-efd7-4b5c-8093-4af09c7dce5c\") " Oct 30 01:55:42.616859 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:42.616523 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-istio-token\" (UniqueName: \"kubernetes.io/projected/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-istio-token\") pod \"3e8eba77-efd7-4b5c-8093-4af09c7dce5c\" (UID: \"3e8eba77-efd7-4b5c-8093-4af09c7dce5c\") " Oct 30 01:55:42.616859 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:42.616625 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-workload-certs\") pod \"3e8eba77-efd7-4b5c-8093-4af09c7dce5c\" (UID: \"3e8eba77-efd7-4b5c-8093-4af09c7dce5c\") " Oct 30 01:55:42.616859 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:42.616682 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-istio-data\") pod \"3e8eba77-efd7-4b5c-8093-4af09c7dce5c\" (UID: \"3e8eba77-efd7-4b5c-8093-4af09c7dce5c\") " Oct 30 01:55:42.616859 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:42.616763 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-credential-socket\") pod \"3e8eba77-efd7-4b5c-8093-4af09c7dce5c\" (UID: \"3e8eba77-efd7-4b5c-8093-4af09c7dce5c\") " Oct 30 01:55:42.616859 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:42.616817 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-istiod-ca-cert\") pod \"3e8eba77-efd7-4b5c-8093-4af09c7dce5c\" (UID: \"3e8eba77-efd7-4b5c-8093-4af09c7dce5c\") " Oct 30 01:55:42.618145 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:42.618110 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-istiod-ca-cert" pod "3e8eba77-efd7-4b5c-8093-4af09c7dce5c" (UID: "3e8eba77-efd7-4b5c-8093-4af09c7dce5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:55:42.618812 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:42.618395 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-workload-certs" pod "3e8eba77-efd7-4b5c-8093-4af09c7dce5c" (UID: "3e8eba77-efd7-4b5c-8093-4af09c7dce5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:55:42.618812 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:42.618600 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-credential-socket" pod "3e8eba77-efd7-4b5c-8093-4af09c7dce5c" (UID: "3e8eba77-efd7-4b5c-8093-4af09c7dce5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:55:42.618812 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:42.618676 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-istio-data" pod "3e8eba77-efd7-4b5c-8093-4af09c7dce5c" (UID: "3e8eba77-efd7-4b5c-8093-4af09c7dce5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:55:42.620733 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:42.619087 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-workload-socket" pod "3e8eba77-efd7-4b5c-8093-4af09c7dce5c" (UID: "3e8eba77-efd7-4b5c-8093-4af09c7dce5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:55:42.621819 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:42.621795 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-istio-token" pod "3e8eba77-efd7-4b5c-8093-4af09c7dce5c" (UID: "3e8eba77-efd7-4b5c-8093-4af09c7dce5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:55:42.621948 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:42.621928 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-istio-envoy" pod "3e8eba77-efd7-4b5c-8093-4af09c7dce5c" (UID: "3e8eba77-efd7-4b5c-8093-4af09c7dce5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:55:42.621993 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:42.621972 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-kube-api-access-k4tsv" pod "3e8eba77-efd7-4b5c-8093-4af09c7dce5c" (UID: "3e8eba77-efd7-4b5c-8093-4af09c7dce5c"). InnerVolumeSpecName "kube-api-access-k4tsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:55:42.622286 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:42.622272 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-istio-podinfo" pod "3e8eba77-efd7-4b5c-8093-4af09c7dce5c" (UID: "3e8eba77-efd7-4b5c-8093-4af09c7dce5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:55:42.650911 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:42.650874 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-0835dd-7f7d4b44f9-zvnd7"] Oct 30 01:55:42.678933 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:42.678904 6221 generic.go:356] "Generic (PLEG): container finished" podID="3e8eba77-efd7-4b5c-8093-4af09c7dce5c" containerID="cc04c43519df2931e1e23a15cc336d7e5f5780a01a26544100645dc3f197d73e" exitCode=0 Oct 30 01:55:42.679418 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:42.678955 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj" Oct 30 01:55:42.679418 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:42.678986 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj" event={"ID":"3e8eba77-efd7-4b5c-8093-4af09c7dce5c","Type":"ContainerDied","Data":"cc04c43519df2931e1e23a15cc336d7e5f5780a01a26544100645dc3f197d73e"} Oct 30 01:55:42.679418 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:42.679051 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj" event={"ID":"3e8eba77-efd7-4b5c-8093-4af09c7dce5c","Type":"ContainerDied","Data":"679852c94cc1535af3f3486998ec5ca1b8ebeaf4f17cc17bdca8652510c18d39"} Oct 30 01:55:42.679418 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:42.679086 6221 scope.go:122] "RemoveContainer" containerID="cc04c43519df2931e1e23a15cc336d7e5f5780a01a26544100645dc3f197d73e" Oct 30 01:55:42.690518 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:42.690501 6221 scope.go:122] "RemoveContainer" containerID="cc04c43519df2931e1e23a15cc336d7e5f5780a01a26544100645dc3f197d73e" Oct 30 01:55:42.690834 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:55:42.690811 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc04c43519df2931e1e23a15cc336d7e5f5780a01a26544100645dc3f197d73e\": container with ID starting with cc04c43519df2931e1e23a15cc336d7e5f5780a01a26544100645dc3f197d73e not found: ID does not exist" containerID="cc04c43519df2931e1e23a15cc336d7e5f5780a01a26544100645dc3f197d73e" Oct 30 01:55:42.690983 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:42.690841 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc04c43519df2931e1e23a15cc336d7e5f5780a01a26544100645dc3f197d73e"} err="failed to get container status \"cc04c43519df2931e1e23a15cc336d7e5f5780a01a26544100645dc3f197d73e\": rpc error: code = NotFound desc = could not find container \"cc04c43519df2931e1e23a15cc336d7e5f5780a01a26544100645dc3f197d73e\": container with ID starting with cc04c43519df2931e1e23a15cc336d7e5f5780a01a26544100645dc3f197d73e not found: ID does not exist" Oct 30 01:55:42.722239 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:42.720727 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:55:42.722239 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:42.720758 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:55:42.722239 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:42.720774 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:55:42.722239 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:42.720788 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:55:42.722239 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:42.720808 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:55:42.722239 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:42.720824 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:55:42.722239 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:42.720841 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:55:42.722239 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:42.720855 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4tsv\" (UniqueName: \"kubernetes.io/projected/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-kube-api-access-k4tsv\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:55:42.722239 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:42.720888 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e8eba77-efd7-4b5c-8093-4af09c7dce5c-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:55:42.748170 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:42.748134 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj"] Oct 30 01:55:42.751381 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:42.751355 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-66scj"] Oct 30 01:55:43.393294 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:43.389989 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf"] Oct 30 01:55:43.393294 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:43.390282 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf" podUID="7b8071ca-a7f0-47a5-aabf-32148fa0463a" containerName="istio-proxy" containerID="cri-o://aabf3dfa503f01f3756aa6c952f83749c93e7e9e6c0152aa1f5c585cc5148fcc" gracePeriod=30 Oct 30 01:55:44.548981 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:44.548941 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e8eba77-efd7-4b5c-8093-4af09c7dce5c" path="/var/lib/kubelet/pods/3e8eba77-efd7-4b5c-8093-4af09c7dce5c/volumes" Oct 30 01:55:46.922559 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:46.920500 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-0835dd/autores-model-0835dd-kserve-7b59855b5-5kjp6" podUID="55b4ef19-8d44-4d6b-95d2-d17c41cd49d6" containerName="main" containerID="cri-o://b9bb827914d2fa1e83daf5178194363df8402693787f16f7f40a8ee3b626ed6f" gracePeriod=45 Oct 30 01:55:47.198313 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:47.198284 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-0835dd/autores-model-0835dd-kserve-7b59855b5-5kjp6" Oct 30 01:55:47.288006 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:47.287972 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55b4ef19-8d44-4d6b-95d2-d17c41cd49d6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/55b4ef19-8d44-4d6b-95d2-d17c41cd49d6-dshm\") pod \"55b4ef19-8d44-4d6b-95d2-d17c41cd49d6\" (UID: \"55b4ef19-8d44-4d6b-95d2-d17c41cd49d6\") " Oct 30 01:55:47.288173 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:47.288140 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55b4ef19-8d44-4d6b-95d2-d17c41cd49d6-tls-certs\" (UniqueName: \"kubernetes.io/secret/55b4ef19-8d44-4d6b-95d2-d17c41cd49d6-tls-certs\") pod \"55b4ef19-8d44-4d6b-95d2-d17c41cd49d6\" (UID: \"55b4ef19-8d44-4d6b-95d2-d17c41cd49d6\") " Oct 30 01:55:47.288226 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:47.288177 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55b4ef19-8d44-4d6b-95d2-d17c41cd49d6-kube-api-access-7bvl8\" (UniqueName: \"kubernetes.io/projected/55b4ef19-8d44-4d6b-95d2-d17c41cd49d6-kube-api-access-7bvl8\") pod \"55b4ef19-8d44-4d6b-95d2-d17c41cd49d6\" (UID: \"55b4ef19-8d44-4d6b-95d2-d17c41cd49d6\") " Oct 30 01:55:47.288226 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:47.288218 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55b4ef19-8d44-4d6b-95d2-d17c41cd49d6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55b4ef19-8d44-4d6b-95d2-d17c41cd49d6-model-cache\") pod \"55b4ef19-8d44-4d6b-95d2-d17c41cd49d6\" (UID: \"55b4ef19-8d44-4d6b-95d2-d17c41cd49d6\") " Oct 30 01:55:47.288395 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:47.288281 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55b4ef19-8d44-4d6b-95d2-d17c41cd49d6-home\" (UniqueName: \"kubernetes.io/empty-dir/55b4ef19-8d44-4d6b-95d2-d17c41cd49d6-home\") pod \"55b4ef19-8d44-4d6b-95d2-d17c41cd49d6\" (UID: \"55b4ef19-8d44-4d6b-95d2-d17c41cd49d6\") " Oct 30 01:55:47.288395 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:47.288349 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55b4ef19-8d44-4d6b-95d2-d17c41cd49d6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55b4ef19-8d44-4d6b-95d2-d17c41cd49d6-tmp-dir\") pod \"55b4ef19-8d44-4d6b-95d2-d17c41cd49d6\" (UID: \"55b4ef19-8d44-4d6b-95d2-d17c41cd49d6\") " Oct 30 01:55:47.288590 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:47.288564 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55b4ef19-8d44-4d6b-95d2-d17c41cd49d6-model-cache" pod "55b4ef19-8d44-4d6b-95d2-d17c41cd49d6" (UID: "55b4ef19-8d44-4d6b-95d2-d17c41cd49d6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:55:47.288684 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:47.288606 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55b4ef19-8d44-4d6b-95d2-d17c41cd49d6-home" pod "55b4ef19-8d44-4d6b-95d2-d17c41cd49d6" (UID: "55b4ef19-8d44-4d6b-95d2-d17c41cd49d6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:55:47.288758 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:47.288734 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55b4ef19-8d44-4d6b-95d2-d17c41cd49d6-tmp-dir" pod "55b4ef19-8d44-4d6b-95d2-d17c41cd49d6" (UID: "55b4ef19-8d44-4d6b-95d2-d17c41cd49d6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:55:47.289082 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:47.289061 6221 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/55b4ef19-8d44-4d6b-95d2-d17c41cd49d6-model-cache\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:55:47.289137 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:47.289089 6221 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/55b4ef19-8d44-4d6b-95d2-d17c41cd49d6-home\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:55:47.289137 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:47.289102 6221 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/55b4ef19-8d44-4d6b-95d2-d17c41cd49d6-tmp-dir\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:55:47.290366 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:47.290343 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55b4ef19-8d44-4d6b-95d2-d17c41cd49d6-dshm" pod "55b4ef19-8d44-4d6b-95d2-d17c41cd49d6" (UID: "55b4ef19-8d44-4d6b-95d2-d17c41cd49d6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:55:47.290461 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:47.290360 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55b4ef19-8d44-4d6b-95d2-d17c41cd49d6-tls-certs" pod "55b4ef19-8d44-4d6b-95d2-d17c41cd49d6" (UID: "55b4ef19-8d44-4d6b-95d2-d17c41cd49d6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 01:55:47.290461 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:47.290407 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55b4ef19-8d44-4d6b-95d2-d17c41cd49d6-kube-api-access-7bvl8" pod "55b4ef19-8d44-4d6b-95d2-d17c41cd49d6" (UID: "55b4ef19-8d44-4d6b-95d2-d17c41cd49d6"). InnerVolumeSpecName "kube-api-access-7bvl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:55:47.390229 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:47.390196 6221 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55b4ef19-8d44-4d6b-95d2-d17c41cd49d6-tls-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:55:47.390229 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:47.390226 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bvl8\" (UniqueName: \"kubernetes.io/projected/55b4ef19-8d44-4d6b-95d2-d17c41cd49d6-kube-api-access-7bvl8\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:55:47.390229 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:47.390235 6221 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/55b4ef19-8d44-4d6b-95d2-d17c41cd49d6-dshm\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:55:47.651612 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:47.651495 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-0835dd-7f7d4b44f9-zvnd7" podUID="b3b7a184-e07b-4bea-a3e1-6d2a5e939f09" containerName="payload-pre-processing" containerID="cri-o://a5a6b2492a27f4a9ade01275fac93f4f78364ff12bd705b94f231b586abf996d" gracePeriod=25 Oct 30 01:55:47.684956 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:47.684929 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-093d23-55b74987dd-8nwrk" Oct 30 01:55:47.716857 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:47.716825 6221 generic.go:356] "Generic (PLEG): container finished" podID="55b4ef19-8d44-4d6b-95d2-d17c41cd49d6" containerID="b9bb827914d2fa1e83daf5178194363df8402693787f16f7f40a8ee3b626ed6f" exitCode=0 Oct 30 01:55:47.717032 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:47.716892 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-0835dd/autores-model-0835dd-kserve-7b59855b5-5kjp6" Oct 30 01:55:47.717032 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:47.716908 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-0835dd/autores-model-0835dd-kserve-7b59855b5-5kjp6" event={"ID":"55b4ef19-8d44-4d6b-95d2-d17c41cd49d6","Type":"ContainerDied","Data":"b9bb827914d2fa1e83daf5178194363df8402693787f16f7f40a8ee3b626ed6f"} Oct 30 01:55:47.717032 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:47.716952 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-0835dd/autores-model-0835dd-kserve-7b59855b5-5kjp6" event={"ID":"55b4ef19-8d44-4d6b-95d2-d17c41cd49d6","Type":"ContainerDied","Data":"5140e0e45592abd66514514d029413ab0a8a11f6e55aa5e2aa68716e183d51c0"} Oct 30 01:55:47.717032 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:47.716968 6221 scope.go:122] "RemoveContainer" containerID="b9bb827914d2fa1e83daf5178194363df8402693787f16f7f40a8ee3b626ed6f" Oct 30 01:55:47.772200 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:47.772171 6221 scope.go:122] "RemoveContainer" containerID="b9bb827914d2fa1e83daf5178194363df8402693787f16f7f40a8ee3b626ed6f" Oct 30 01:55:47.773495 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:55:47.772631 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9bb827914d2fa1e83daf5178194363df8402693787f16f7f40a8ee3b626ed6f\": container with ID starting with b9bb827914d2fa1e83daf5178194363df8402693787f16f7f40a8ee3b626ed6f not found: ID does not exist" containerID="b9bb827914d2fa1e83daf5178194363df8402693787f16f7f40a8ee3b626ed6f" Oct 30 01:55:47.773495 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:47.772675 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9bb827914d2fa1e83daf5178194363df8402693787f16f7f40a8ee3b626ed6f"} err="failed to get container status \"b9bb827914d2fa1e83daf5178194363df8402693787f16f7f40a8ee3b626ed6f\": rpc error: code = NotFound desc = could not find container \"b9bb827914d2fa1e83daf5178194363df8402693787f16f7f40a8ee3b626ed6f\": container with ID starting with b9bb827914d2fa1e83daf5178194363df8402693787f16f7f40a8ee3b626ed6f not found: ID does not exist" Oct 30 01:55:47.778424 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:47.778396 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-0835dd/autores-model-0835dd-kserve-7b59855b5-5kjp6"] Oct 30 01:55:47.784525 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:47.784496 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-0835dd/autores-model-0835dd-kserve-7b59855b5-5kjp6"] Oct 30 01:55:47.948424 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:47.948401 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-0835dd-7f7d4b44f9-zvnd7" Oct 30 01:55:47.999371 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:47.999334 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3b7a184-e07b-4bea-a3e1-6d2a5e939f09-kube-api-access-qsch5\" (UniqueName: \"kubernetes.io/projected/b3b7a184-e07b-4bea-a3e1-6d2a5e939f09-kube-api-access-qsch5\") pod \"b3b7a184-e07b-4bea-a3e1-6d2a5e939f09\" (UID: \"b3b7a184-e07b-4bea-a3e1-6d2a5e939f09\") " Oct 30 01:55:47.999587 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:47.999478 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3b7a184-e07b-4bea-a3e1-6d2a5e939f09-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3b7a184-e07b-4bea-a3e1-6d2a5e939f09-plugins-config-volume\") pod \"b3b7a184-e07b-4bea-a3e1-6d2a5e939f09\" (UID: \"b3b7a184-e07b-4bea-a3e1-6d2a5e939f09\") " Oct 30 01:55:48.000192 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.000163 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3b7a184-e07b-4bea-a3e1-6d2a5e939f09-plugins-config-volume" pod "b3b7a184-e07b-4bea-a3e1-6d2a5e939f09" (UID: "b3b7a184-e07b-4bea-a3e1-6d2a5e939f09"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:55:48.000659 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.000636 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3b7a184-e07b-4bea-a3e1-6d2a5e939f09-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:55:48.002602 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.002575 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3b7a184-e07b-4bea-a3e1-6d2a5e939f09-kube-api-access-qsch5" pod "b3b7a184-e07b-4bea-a3e1-6d2a5e939f09" (UID: "b3b7a184-e07b-4bea-a3e1-6d2a5e939f09"). InnerVolumeSpecName "kube-api-access-qsch5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:55:48.102069 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.102035 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsch5\" (UniqueName: \"kubernetes.io/projected/b3b7a184-e07b-4bea-a3e1-6d2a5e939f09-kube-api-access-qsch5\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:55:48.549792 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.549747 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55b4ef19-8d44-4d6b-95d2-d17c41cd49d6" path="/var/lib/kubelet/pods/55b4ef19-8d44-4d6b-95d2-d17c41cd49d6/volumes" Oct 30 01:55:48.665120 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.665048 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf" Oct 30 01:55:48.710852 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.710818 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b8071ca-a7f0-47a5-aabf-32148fa0463a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b8071ca-a7f0-47a5-aabf-32148fa0463a-workload-certs\") pod \"7b8071ca-a7f0-47a5-aabf-32148fa0463a\" (UID: \"7b8071ca-a7f0-47a5-aabf-32148fa0463a\") " Oct 30 01:55:48.711016 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.710881 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b8071ca-a7f0-47a5-aabf-32148fa0463a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b8071ca-a7f0-47a5-aabf-32148fa0463a-istio-podinfo\") pod \"7b8071ca-a7f0-47a5-aabf-32148fa0463a\" (UID: \"7b8071ca-a7f0-47a5-aabf-32148fa0463a\") " Oct 30 01:55:48.711016 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.710983 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b8071ca-a7f0-47a5-aabf-32148fa0463a-kube-api-access-jzg6f\" (UniqueName: \"kubernetes.io/projected/7b8071ca-a7f0-47a5-aabf-32148fa0463a-kube-api-access-jzg6f\") pod \"7b8071ca-a7f0-47a5-aabf-32148fa0463a\" (UID: \"7b8071ca-a7f0-47a5-aabf-32148fa0463a\") " Oct 30 01:55:48.711126 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.711039 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b8071ca-a7f0-47a5-aabf-32148fa0463a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b8071ca-a7f0-47a5-aabf-32148fa0463a-istiod-ca-cert\") pod \"7b8071ca-a7f0-47a5-aabf-32148fa0463a\" (UID: \"7b8071ca-a7f0-47a5-aabf-32148fa0463a\") " Oct 30 01:55:48.711126 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.711072 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b8071ca-a7f0-47a5-aabf-32148fa0463a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b8071ca-a7f0-47a5-aabf-32148fa0463a-workload-socket\") pod \"7b8071ca-a7f0-47a5-aabf-32148fa0463a\" (UID: \"7b8071ca-a7f0-47a5-aabf-32148fa0463a\") " Oct 30 01:55:48.711126 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.711101 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b8071ca-a7f0-47a5-aabf-32148fa0463a-workload-certs" pod "7b8071ca-a7f0-47a5-aabf-32148fa0463a" (UID: "7b8071ca-a7f0-47a5-aabf-32148fa0463a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:55:48.711291 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.711154 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b8071ca-a7f0-47a5-aabf-32148fa0463a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b8071ca-a7f0-47a5-aabf-32148fa0463a-istio-data\") pod \"7b8071ca-a7f0-47a5-aabf-32148fa0463a\" (UID: \"7b8071ca-a7f0-47a5-aabf-32148fa0463a\") " Oct 30 01:55:48.711291 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.711242 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b8071ca-a7f0-47a5-aabf-32148fa0463a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b8071ca-a7f0-47a5-aabf-32148fa0463a-credential-socket\") pod \"7b8071ca-a7f0-47a5-aabf-32148fa0463a\" (UID: \"7b8071ca-a7f0-47a5-aabf-32148fa0463a\") " Oct 30 01:55:48.711397 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.711312 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b8071ca-a7f0-47a5-aabf-32148fa0463a-istio-token\" (UniqueName: \"kubernetes.io/projected/7b8071ca-a7f0-47a5-aabf-32148fa0463a-istio-token\") pod \"7b8071ca-a7f0-47a5-aabf-32148fa0463a\" (UID: \"7b8071ca-a7f0-47a5-aabf-32148fa0463a\") " Oct 30 01:55:48.711397 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.711359 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b8071ca-a7f0-47a5-aabf-32148fa0463a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b8071ca-a7f0-47a5-aabf-32148fa0463a-istio-envoy\") pod \"7b8071ca-a7f0-47a5-aabf-32148fa0463a\" (UID: \"7b8071ca-a7f0-47a5-aabf-32148fa0463a\") " Oct 30 01:55:48.711397 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.711372 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b8071ca-a7f0-47a5-aabf-32148fa0463a-workload-socket" pod "7b8071ca-a7f0-47a5-aabf-32148fa0463a" (UID: "7b8071ca-a7f0-47a5-aabf-32148fa0463a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:55:48.711564 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.711503 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b8071ca-a7f0-47a5-aabf-32148fa0463a-credential-socket" pod "7b8071ca-a7f0-47a5-aabf-32148fa0463a" (UID: "7b8071ca-a7f0-47a5-aabf-32148fa0463a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:55:48.711564 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.711441 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b8071ca-a7f0-47a5-aabf-32148fa0463a-istiod-ca-cert" pod "7b8071ca-a7f0-47a5-aabf-32148fa0463a" (UID: "7b8071ca-a7f0-47a5-aabf-32148fa0463a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:55:48.711564 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.711518 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b8071ca-a7f0-47a5-aabf-32148fa0463a-istio-data" pod "7b8071ca-a7f0-47a5-aabf-32148fa0463a" (UID: "7b8071ca-a7f0-47a5-aabf-32148fa0463a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:55:48.711999 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.711970 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b8071ca-a7f0-47a5-aabf-32148fa0463a-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:55:48.711999 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.712002 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b8071ca-a7f0-47a5-aabf-32148fa0463a-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:55:48.712182 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.712016 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b8071ca-a7f0-47a5-aabf-32148fa0463a-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:55:48.712182 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.712030 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b8071ca-a7f0-47a5-aabf-32148fa0463a-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:55:48.712182 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.712044 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b8071ca-a7f0-47a5-aabf-32148fa0463a-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:55:48.713297 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.713273 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b8071ca-a7f0-47a5-aabf-32148fa0463a-kube-api-access-jzg6f" pod "7b8071ca-a7f0-47a5-aabf-32148fa0463a" (UID: "7b8071ca-a7f0-47a5-aabf-32148fa0463a"). InnerVolumeSpecName "kube-api-access-jzg6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:55:48.713413 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.713395 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b8071ca-a7f0-47a5-aabf-32148fa0463a-istio-token" pod "7b8071ca-a7f0-47a5-aabf-32148fa0463a" (UID: "7b8071ca-a7f0-47a5-aabf-32148fa0463a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:55:48.713472 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.713458 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b8071ca-a7f0-47a5-aabf-32148fa0463a-istio-envoy" pod "7b8071ca-a7f0-47a5-aabf-32148fa0463a" (UID: "7b8071ca-a7f0-47a5-aabf-32148fa0463a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 01:55:48.713520 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.713480 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b8071ca-a7f0-47a5-aabf-32148fa0463a-istio-podinfo" pod "7b8071ca-a7f0-47a5-aabf-32148fa0463a" (UID: "7b8071ca-a7f0-47a5-aabf-32148fa0463a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 01:55:48.728043 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.728012 6221 generic.go:356] "Generic (PLEG): container finished" podID="b3b7a184-e07b-4bea-a3e1-6d2a5e939f09" containerID="a5a6b2492a27f4a9ade01275fac93f4f78364ff12bd705b94f231b586abf996d" exitCode=0 Oct 30 01:55:48.728187 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.728054 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-0835dd-7f7d4b44f9-zvnd7" Oct 30 01:55:48.728187 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.728097 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-0835dd-7f7d4b44f9-zvnd7" event={"ID":"b3b7a184-e07b-4bea-a3e1-6d2a5e939f09","Type":"ContainerDied","Data":"a5a6b2492a27f4a9ade01275fac93f4f78364ff12bd705b94f231b586abf996d"} Oct 30 01:55:48.728187 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.728162 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-0835dd-7f7d4b44f9-zvnd7" event={"ID":"b3b7a184-e07b-4bea-a3e1-6d2a5e939f09","Type":"ContainerDied","Data":"64d1daef48e55c67dfa9aef8b922b74234ef84a6a403db4aba27576297c8362d"} Oct 30 01:55:48.728350 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.728194 6221 scope.go:122] "RemoveContainer" containerID="a5a6b2492a27f4a9ade01275fac93f4f78364ff12bd705b94f231b586abf996d" Oct 30 01:55:48.730893 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.730642 6221 generic.go:356] "Generic (PLEG): container finished" podID="7b8071ca-a7f0-47a5-aabf-32148fa0463a" containerID="aabf3dfa503f01f3756aa6c952f83749c93e7e9e6c0152aa1f5c585cc5148fcc" exitCode=0 Oct 30 01:55:48.730893 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.730799 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf" event={"ID":"7b8071ca-a7f0-47a5-aabf-32148fa0463a","Type":"ContainerDied","Data":"aabf3dfa503f01f3756aa6c952f83749c93e7e9e6c0152aa1f5c585cc5148fcc"} Oct 30 01:55:48.730893 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.730890 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf" event={"ID":"7b8071ca-a7f0-47a5-aabf-32148fa0463a","Type":"ContainerDied","Data":"eb0c7e1517808af08425d271f564d4a02517fc28da4c4a0e113b74ff10e5b09b"} Oct 30 01:55:48.732515 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.730941 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf" Oct 30 01:55:48.742577 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.742520 6221 scope.go:122] "RemoveContainer" containerID="a5a6b2492a27f4a9ade01275fac93f4f78364ff12bd705b94f231b586abf996d" Oct 30 01:55:48.742985 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:55:48.742954 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5a6b2492a27f4a9ade01275fac93f4f78364ff12bd705b94f231b586abf996d\": container with ID starting with a5a6b2492a27f4a9ade01275fac93f4f78364ff12bd705b94f231b586abf996d not found: ID does not exist" containerID="a5a6b2492a27f4a9ade01275fac93f4f78364ff12bd705b94f231b586abf996d" Oct 30 01:55:48.743098 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.742995 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5a6b2492a27f4a9ade01275fac93f4f78364ff12bd705b94f231b586abf996d"} err="failed to get container status \"a5a6b2492a27f4a9ade01275fac93f4f78364ff12bd705b94f231b586abf996d\": rpc error: code = NotFound desc = could not find container \"a5a6b2492a27f4a9ade01275fac93f4f78364ff12bd705b94f231b586abf996d\": container with ID starting with a5a6b2492a27f4a9ade01275fac93f4f78364ff12bd705b94f231b586abf996d not found: ID does not exist" Oct 30 01:55:48.743098 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.743018 6221 scope.go:122] "RemoveContainer" containerID="aabf3dfa503f01f3756aa6c952f83749c93e7e9e6c0152aa1f5c585cc5148fcc" Oct 30 01:55:48.755985 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.755963 6221 scope.go:122] "RemoveContainer" containerID="aabf3dfa503f01f3756aa6c952f83749c93e7e9e6c0152aa1f5c585cc5148fcc" Oct 30 01:55:48.756296 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:55:48.756277 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aabf3dfa503f01f3756aa6c952f83749c93e7e9e6c0152aa1f5c585cc5148fcc\": container with ID starting with aabf3dfa503f01f3756aa6c952f83749c93e7e9e6c0152aa1f5c585cc5148fcc not found: ID does not exist" containerID="aabf3dfa503f01f3756aa6c952f83749c93e7e9e6c0152aa1f5c585cc5148fcc" Oct 30 01:55:48.756387 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.756304 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aabf3dfa503f01f3756aa6c952f83749c93e7e9e6c0152aa1f5c585cc5148fcc"} err="failed to get container status \"aabf3dfa503f01f3756aa6c952f83749c93e7e9e6c0152aa1f5c585cc5148fcc\": rpc error: code = NotFound desc = could not find container \"aabf3dfa503f01f3756aa6c952f83749c93e7e9e6c0152aa1f5c585cc5148fcc\": container with ID starting with aabf3dfa503f01f3756aa6c952f83749c93e7e9e6c0152aa1f5c585cc5148fcc not found: ID does not exist" Oct 30 01:55:48.782398 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.782365 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf"] Oct 30 01:55:48.787134 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.787109 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-6cgtf"] Oct 30 01:55:48.813083 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.813053 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzg6f\" (UniqueName: \"kubernetes.io/projected/7b8071ca-a7f0-47a5-aabf-32148fa0463a-kube-api-access-jzg6f\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:55:48.813083 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.813081 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b8071ca-a7f0-47a5-aabf-32148fa0463a-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:55:48.813239 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.813090 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b8071ca-a7f0-47a5-aabf-32148fa0463a-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:55:48.813239 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.813099 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b8071ca-a7f0-47a5-aabf-32148fa0463a-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:55:48.822610 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.822585 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-0835dd-7f7d4b44f9-zvnd7"] Oct 30 01:55:48.825901 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:48.825880 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-0835dd-7f7d4b44f9-zvnd7"] Oct 30 01:55:49.894371 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:49.894338 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd8173-7b89fb6b85-rw4zt"] Oct 30 01:55:49.895052 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:49.895037 6221 cpu_manager.go:414] "containerMap: removing container" podUID="3e8eba77-efd7-4b5c-8093-4af09c7dce5c" containerName="istio-proxy" Oct 30 01:55:49.895112 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:49.895053 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e8eba77-efd7-4b5c-8093-4af09c7dce5c" containerName="istio-proxy" Oct 30 01:55:49.895112 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:49.895070 6221 cpu_manager.go:414] "containerMap: removing container" podUID="7b8071ca-a7f0-47a5-aabf-32148fa0463a" containerName="istio-proxy" Oct 30 01:55:49.895112 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:49.895075 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b8071ca-a7f0-47a5-aabf-32148fa0463a" containerName="istio-proxy" Oct 30 01:55:49.895112 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:49.895086 6221 cpu_manager.go:414] "containerMap: removing container" podUID="55b4ef19-8d44-4d6b-95d2-d17c41cd49d6" containerName="main" Oct 30 01:55:49.895112 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:49.895091 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55b4ef19-8d44-4d6b-95d2-d17c41cd49d6" containerName="main" Oct 30 01:55:49.895112 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:49.895104 6221 cpu_manager.go:414] "containerMap: removing container" podUID="b3b7a184-e07b-4bea-a3e1-6d2a5e939f09" containerName="payload-pre-processing" Oct 30 01:55:49.895112 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:49.895110 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3b7a184-e07b-4bea-a3e1-6d2a5e939f09" containerName="payload-pre-processing" Oct 30 01:55:49.895324 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:49.895208 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e8eba77-efd7-4b5c-8093-4af09c7dce5c" containerName="istio-proxy" Oct 30 01:55:49.895324 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:49.895216 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b8071ca-a7f0-47a5-aabf-32148fa0463a" containerName="istio-proxy" Oct 30 01:55:49.895324 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:49.895236 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="55b4ef19-8d44-4d6b-95d2-d17c41cd49d6" containerName="main" Oct 30 01:55:49.895324 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:49.895246 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3b7a184-e07b-4bea-a3e1-6d2a5e939f09" containerName="payload-pre-processing" Oct 30 01:55:49.900264 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:49.900242 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd8173-7b89fb6b85-rw4zt" Oct 30 01:55:49.917657 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:49.917624 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-cd8173-serving-cert\"" Oct 30 01:55:49.935426 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:49.935394 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd8173-7b89fb6b85-rw4zt"] Oct 30 01:55:50.029106 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:50.029067 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0eec39a3-c8c5-47fd-b1c3-132da72e7b65-maas-api-tls\") pod \"maas-api-e2e-extmod-b-cd8173-7b89fb6b85-rw4zt\" (UID: \"0eec39a3-c8c5-47fd-b1c3-132da72e7b65\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd8173-7b89fb6b85-rw4zt" Oct 30 01:55:50.029106 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:50.029111 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0eec39a3-c8c5-47fd-b1c3-132da72e7b65-metrics-certs\") pod \"maas-api-e2e-extmod-b-cd8173-7b89fb6b85-rw4zt\" (UID: \"0eec39a3-c8c5-47fd-b1c3-132da72e7b65\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd8173-7b89fb6b85-rw4zt" Oct 30 01:55:50.029379 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:50.029186 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b899z\" (UniqueName: \"kubernetes.io/projected/0eec39a3-c8c5-47fd-b1c3-132da72e7b65-kube-api-access-b899z\") pod \"maas-api-e2e-extmod-b-cd8173-7b89fb6b85-rw4zt\" (UID: \"0eec39a3-c8c5-47fd-b1c3-132da72e7b65\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd8173-7b89fb6b85-rw4zt" Oct 30 01:55:50.029448 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:50.029412 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eec39a3-c8c5-47fd-b1c3-132da72e7b65-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-cd8173-7b89fb6b85-rw4zt\" (UID: \"0eec39a3-c8c5-47fd-b1c3-132da72e7b65\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd8173-7b89fb6b85-rw4zt" Oct 30 01:55:50.130605 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:50.130564 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eec39a3-c8c5-47fd-b1c3-132da72e7b65-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-cd8173-7b89fb6b85-rw4zt\" (UID: \"0eec39a3-c8c5-47fd-b1c3-132da72e7b65\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd8173-7b89fb6b85-rw4zt" Oct 30 01:55:50.130787 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:50.130694 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0eec39a3-c8c5-47fd-b1c3-132da72e7b65-maas-api-tls\") pod \"maas-api-e2e-extmod-b-cd8173-7b89fb6b85-rw4zt\" (UID: \"0eec39a3-c8c5-47fd-b1c3-132da72e7b65\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd8173-7b89fb6b85-rw4zt" Oct 30 01:55:50.130787 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:50.130730 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0eec39a3-c8c5-47fd-b1c3-132da72e7b65-metrics-certs\") pod \"maas-api-e2e-extmod-b-cd8173-7b89fb6b85-rw4zt\" (UID: \"0eec39a3-c8c5-47fd-b1c3-132da72e7b65\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd8173-7b89fb6b85-rw4zt" Oct 30 01:55:50.130787 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:50.130765 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b899z\" (UniqueName: \"kubernetes.io/projected/0eec39a3-c8c5-47fd-b1c3-132da72e7b65-kube-api-access-b899z\") pod \"maas-api-e2e-extmod-b-cd8173-7b89fb6b85-rw4zt\" (UID: \"0eec39a3-c8c5-47fd-b1c3-132da72e7b65\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd8173-7b89fb6b85-rw4zt" Oct 30 01:55:50.131284 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:50.131263 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eec39a3-c8c5-47fd-b1c3-132da72e7b65-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-cd8173-7b89fb6b85-rw4zt\" (UID: \"0eec39a3-c8c5-47fd-b1c3-132da72e7b65\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd8173-7b89fb6b85-rw4zt" Oct 30 01:55:50.133300 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:50.133283 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0eec39a3-c8c5-47fd-b1c3-132da72e7b65-metrics-certs\") pod \"maas-api-e2e-extmod-b-cd8173-7b89fb6b85-rw4zt\" (UID: \"0eec39a3-c8c5-47fd-b1c3-132da72e7b65\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd8173-7b89fb6b85-rw4zt" Oct 30 01:55:50.133419 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:50.133396 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0eec39a3-c8c5-47fd-b1c3-132da72e7b65-maas-api-tls\") pod \"maas-api-e2e-extmod-b-cd8173-7b89fb6b85-rw4zt\" (UID: \"0eec39a3-c8c5-47fd-b1c3-132da72e7b65\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd8173-7b89fb6b85-rw4zt" Oct 30 01:55:50.170766 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:50.170696 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b899z\" (UniqueName: \"kubernetes.io/projected/0eec39a3-c8c5-47fd-b1c3-132da72e7b65-kube-api-access-b899z\") pod \"maas-api-e2e-extmod-b-cd8173-7b89fb6b85-rw4zt\" (UID: \"0eec39a3-c8c5-47fd-b1c3-132da72e7b65\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd8173-7b89fb6b85-rw4zt" Oct 30 01:55:50.214784 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:50.214750 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd8173-7b89fb6b85-rw4zt" Oct 30 01:55:50.517973 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:50.517944 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd8173-7b89fb6b85-rw4zt"] Oct 30 01:55:50.520028 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:55:50.519999 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eec39a3_c8c5_47fd_b1c3_132da72e7b65.slice/crio-3921a026fb0dde637d46e72a172aeda3ae3056d71b552edce4da61f4586a6f93 WatchSource:0}: Error finding container 3921a026fb0dde637d46e72a172aeda3ae3056d71b552edce4da61f4586a6f93: Status 404 returned error can't find the container with id 3921a026fb0dde637d46e72a172aeda3ae3056d71b552edce4da61f4586a6f93 Oct 30 01:55:50.548767 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:50.548738 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b8071ca-a7f0-47a5-aabf-32148fa0463a" path="/var/lib/kubelet/pods/7b8071ca-a7f0-47a5-aabf-32148fa0463a/volumes" Oct 30 01:55:50.549236 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:50.549216 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3b7a184-e07b-4bea-a3e1-6d2a5e939f09" path="/var/lib/kubelet/pods/b3b7a184-e07b-4bea-a3e1-6d2a5e939f09/volumes" Oct 30 01:55:50.750683 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:50.750634 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd8173-7b89fb6b85-rw4zt" event={"ID":"0eec39a3-c8c5-47fd-b1c3-132da72e7b65","Type":"ContainerStarted","Data":"3921a026fb0dde637d46e72a172aeda3ae3056d71b552edce4da61f4586a6f93"} Oct 30 01:55:51.495009 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:55:51.494968 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:55:51.759233 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:51.759115 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd8173-7b89fb6b85-rw4zt" event={"ID":"0eec39a3-c8c5-47fd-b1c3-132da72e7b65","Type":"ContainerStarted","Data":"59e292a5bc4003538ee23a517aa00cd0dab55115b8d8abc739b40e7d9314f84e"} Oct 30 01:55:51.759444 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:51.759423 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd8173-7b89fb6b85-rw4zt" Oct 30 01:55:57.773301 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:57.773272 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd8173-7b89fb6b85-rw4zt" Oct 30 01:55:57.817903 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:57.817843 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd8173-7b89fb6b85-rw4zt" podStartSLOduration=8.470356068 podStartE2EDuration="8.8178281s" totalImagesPullingTime="347.472032ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:55:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:55:51.817958969 +0000 UTC m=+2275.396246327" watchObservedRunningTime="2026-10-30 01:55:57.8178281 +0000 UTC m=+2281.396115436" Oct 30 01:55:58.138419 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:55:58.138382 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:56:21.495627 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:56:21.495589 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:56:28.100963 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:56:28.100931 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76464j85z"] Oct 30 01:56:28.105489 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:56:28.105465 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76464j85z" Oct 30 01:56:28.120785 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:56:28.120746 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-093d23\"/\"openshift-service-ca.crt\"" Oct 30 01:56:28.120785 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:56:28.120781 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-093d23\"/\"agc-cred-8b0342fa1e4d\"" Oct 30 01:56:28.121584 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:56:28.121522 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76464j85z"] Oct 30 01:56:28.130291 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:56:28.130265 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-093d23\"/\"kube-root-ca.crt\"" Oct 30 01:56:28.130437 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:56:28.130323 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-093d23\"/\"payload-processing-external-model-plugins-e2e-extmod-b-093d23\"" Oct 30 01:56:28.130508 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:56:28.130458 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-093d23\"/\"routing-overlay\"" Oct 30 01:56:28.130591 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:56:28.130557 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-093d23\"/\"payload-processing-external-model-e2e--4282dfa5-dockercfg-q42pg\"" Oct 30 01:56:28.236575 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:56:28.232623 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1db9df6e-e021-443b-8c97-139ccfc7c1cf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-76464j85z\" (UID: \"1db9df6e-e021-443b-8c97-139ccfc7c1cf\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76464j85z" Oct 30 01:56:28.236575 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:56:28.232814 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjz5c\" (UniqueName: \"kubernetes.io/projected/1db9df6e-e021-443b-8c97-139ccfc7c1cf-kube-api-access-cjz5c\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-76464j85z\" (UID: \"1db9df6e-e021-443b-8c97-139ccfc7c1cf\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76464j85z" Oct 30 01:56:28.236575 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:56:28.232848 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1db9df6e-e021-443b-8c97-139ccfc7c1cf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-76464j85z\" (UID: \"1db9df6e-e021-443b-8c97-139ccfc7c1cf\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76464j85z" Oct 30 01:56:28.236575 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:56:28.232896 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1db9df6e-e021-443b-8c97-139ccfc7c1cf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-76464j85z\" (UID: \"1db9df6e-e021-443b-8c97-139ccfc7c1cf\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76464j85z" Oct 30 01:56:28.335639 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:56:28.335605 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1db9df6e-e021-443b-8c97-139ccfc7c1cf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-76464j85z\" (UID: \"1db9df6e-e021-443b-8c97-139ccfc7c1cf\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76464j85z" Oct 30 01:56:28.335828 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:56:28.335740 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjz5c\" (UniqueName: \"kubernetes.io/projected/1db9df6e-e021-443b-8c97-139ccfc7c1cf-kube-api-access-cjz5c\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-76464j85z\" (UID: \"1db9df6e-e021-443b-8c97-139ccfc7c1cf\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76464j85z" Oct 30 01:56:28.335828 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:56:28.335771 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1db9df6e-e021-443b-8c97-139ccfc7c1cf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-76464j85z\" (UID: \"1db9df6e-e021-443b-8c97-139ccfc7c1cf\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76464j85z" Oct 30 01:56:28.335965 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:56:28.335946 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1db9df6e-e021-443b-8c97-139ccfc7c1cf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-76464j85z\" (UID: \"1db9df6e-e021-443b-8c97-139ccfc7c1cf\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76464j85z" Oct 30 01:56:28.336576 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:56:28.336508 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1db9df6e-e021-443b-8c97-139ccfc7c1cf-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-76464j85z\" (UID: \"1db9df6e-e021-443b-8c97-139ccfc7c1cf\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76464j85z" Oct 30 01:56:28.336723 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:56:28.336609 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1db9df6e-e021-443b-8c97-139ccfc7c1cf-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-76464j85z\" (UID: \"1db9df6e-e021-443b-8c97-139ccfc7c1cf\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76464j85z" Oct 30 01:56:28.338275 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:56:28.338254 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1db9df6e-e021-443b-8c97-139ccfc7c1cf-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-76464j85z\" (UID: \"1db9df6e-e021-443b-8c97-139ccfc7c1cf\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76464j85z" Oct 30 01:56:28.355551 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:56:28.355472 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjz5c\" (UniqueName: \"kubernetes.io/projected/1db9df6e-e021-443b-8c97-139ccfc7c1cf-kube-api-access-cjz5c\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-76464j85z\" (UID: \"1db9df6e-e021-443b-8c97-139ccfc7c1cf\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76464j85z" Oct 30 01:56:28.420147 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:56:28.420099 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76464j85z" Oct 30 01:56:28.806240 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:56:28.806204 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76464j85z"] Oct 30 01:56:28.867617 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:56:28.865576 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1db9df6e_e021_443b_8c97_139ccfc7c1cf.slice/crio-5b9d9425569796e277575bd21f14ae04489763d623b6a414d15679a170364c1f WatchSource:0}: Error finding container 5b9d9425569796e277575bd21f14ae04489763d623b6a414d15679a170364c1f: Status 404 returned error can't find the container with id 5b9d9425569796e277575bd21f14ae04489763d623b6a414d15679a170364c1f Oct 30 01:56:29.101917 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:56:29.101829 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76464j85z" event={"ID":"1db9df6e-e021-443b-8c97-139ccfc7c1cf","Type":"ContainerStarted","Data":"b021cde7fd444ab7db6dd7857a3152ea1dc9fbba4c22beb96b5551669e13733a"} Oct 30 01:56:29.101917 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:56:29.101889 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76464j85z" event={"ID":"1db9df6e-e021-443b-8c97-139ccfc7c1cf","Type":"ContainerStarted","Data":"5b9d9425569796e277575bd21f14ae04489763d623b6a414d15679a170364c1f"} Oct 30 01:56:29.102349 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:56:29.102061 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76464j85z" Oct 30 01:56:29.147415 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:56:29.147318 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76464j85z" podStartSLOduration=1.147300237 podStartE2EDuration="1.147300237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:56:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:56:29.142007678 +0000 UTC m=+2312.720295014" watchObservedRunningTime="2026-10-30 01:56:29.147300237 +0000 UTC m=+2312.725587574" Oct 30 01:56:32.043466 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:56:32.043412 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:56:40.112202 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:56:40.112172 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76464j85z" Oct 30 01:56:51.494666 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:56:51.494611 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:57:04.638602 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:57:04.638565 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:57:11.155645 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:57:11.155561 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-817fd7-764dbbc7f9-9fsbh"] Oct 30 01:57:11.161221 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:57:11.161198 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-817fd7-764dbbc7f9-9fsbh" Oct 30 01:57:11.177179 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:57:11.177147 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-817fd7-764dbbc7f9-9fsbh"] Oct 30 01:57:11.187570 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:57:11.187515 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-817fd7-serving-cert\"" Oct 30 01:57:11.219331 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:57:11.219299 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8de244f-5cf0-4543-9118-6684c2be86f0-maas-api-tls\") pod \"maas-api-e2e-extmod-a-817fd7-764dbbc7f9-9fsbh\" (UID: \"f8de244f-5cf0-4543-9118-6684c2be86f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-817fd7-764dbbc7f9-9fsbh" Oct 30 01:57:11.219496 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:57:11.219343 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8de244f-5cf0-4543-9118-6684c2be86f0-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-817fd7-764dbbc7f9-9fsbh\" (UID: \"f8de244f-5cf0-4543-9118-6684c2be86f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-817fd7-764dbbc7f9-9fsbh" Oct 30 01:57:11.219560 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:57:11.219506 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8de244f-5cf0-4543-9118-6684c2be86f0-metrics-certs\") pod \"maas-api-e2e-extmod-a-817fd7-764dbbc7f9-9fsbh\" (UID: \"f8de244f-5cf0-4543-9118-6684c2be86f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-817fd7-764dbbc7f9-9fsbh" Oct 30 01:57:11.219610 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:57:11.219592 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96d5w\" (UniqueName: \"kubernetes.io/projected/f8de244f-5cf0-4543-9118-6684c2be86f0-kube-api-access-96d5w\") pod \"maas-api-e2e-extmod-a-817fd7-764dbbc7f9-9fsbh\" (UID: \"f8de244f-5cf0-4543-9118-6684c2be86f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-817fd7-764dbbc7f9-9fsbh" Oct 30 01:57:11.321209 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:57:11.321173 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8de244f-5cf0-4543-9118-6684c2be86f0-maas-api-tls\") pod \"maas-api-e2e-extmod-a-817fd7-764dbbc7f9-9fsbh\" (UID: \"f8de244f-5cf0-4543-9118-6684c2be86f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-817fd7-764dbbc7f9-9fsbh" Oct 30 01:57:11.321389 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:57:11.321222 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8de244f-5cf0-4543-9118-6684c2be86f0-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-817fd7-764dbbc7f9-9fsbh\" (UID: \"f8de244f-5cf0-4543-9118-6684c2be86f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-817fd7-764dbbc7f9-9fsbh" Oct 30 01:57:11.321389 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:57:11.321309 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8de244f-5cf0-4543-9118-6684c2be86f0-metrics-certs\") pod \"maas-api-e2e-extmod-a-817fd7-764dbbc7f9-9fsbh\" (UID: \"f8de244f-5cf0-4543-9118-6684c2be86f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-817fd7-764dbbc7f9-9fsbh" Oct 30 01:57:11.321389 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:57:11.321362 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96d5w\" (UniqueName: \"kubernetes.io/projected/f8de244f-5cf0-4543-9118-6684c2be86f0-kube-api-access-96d5w\") pod \"maas-api-e2e-extmod-a-817fd7-764dbbc7f9-9fsbh\" (UID: \"f8de244f-5cf0-4543-9118-6684c2be86f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-817fd7-764dbbc7f9-9fsbh" Oct 30 01:57:11.322257 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:57:11.322224 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8de244f-5cf0-4543-9118-6684c2be86f0-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-817fd7-764dbbc7f9-9fsbh\" (UID: \"f8de244f-5cf0-4543-9118-6684c2be86f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-817fd7-764dbbc7f9-9fsbh" Oct 30 01:57:11.323982 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:57:11.323959 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8de244f-5cf0-4543-9118-6684c2be86f0-metrics-certs\") pod \"maas-api-e2e-extmod-a-817fd7-764dbbc7f9-9fsbh\" (UID: \"f8de244f-5cf0-4543-9118-6684c2be86f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-817fd7-764dbbc7f9-9fsbh" Oct 30 01:57:11.324089 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:57:11.324007 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8de244f-5cf0-4543-9118-6684c2be86f0-maas-api-tls\") pod \"maas-api-e2e-extmod-a-817fd7-764dbbc7f9-9fsbh\" (UID: \"f8de244f-5cf0-4543-9118-6684c2be86f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-817fd7-764dbbc7f9-9fsbh" Oct 30 01:57:11.341210 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:57:11.341176 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96d5w\" (UniqueName: \"kubernetes.io/projected/f8de244f-5cf0-4543-9118-6684c2be86f0-kube-api-access-96d5w\") pod \"maas-api-e2e-extmod-a-817fd7-764dbbc7f9-9fsbh\" (UID: \"f8de244f-5cf0-4543-9118-6684c2be86f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-817fd7-764dbbc7f9-9fsbh" Oct 30 01:57:11.476022 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:57:11.475939 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-817fd7-764dbbc7f9-9fsbh" Oct 30 01:57:11.783651 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:57:11.783620 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-817fd7-764dbbc7f9-9fsbh"] Oct 30 01:57:11.785244 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:57:11.785219 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8de244f_5cf0_4543_9118_6684c2be86f0.slice/crio-180620a2f20cd31938232d1cdf2c6db1d14aa4e6c72c3d0fdd1290bfd5ac68cd WatchSource:0}: Error finding container 180620a2f20cd31938232d1cdf2c6db1d14aa4e6c72c3d0fdd1290bfd5ac68cd: Status 404 returned error can't find the container with id 180620a2f20cd31938232d1cdf2c6db1d14aa4e6c72c3d0fdd1290bfd5ac68cd Oct 30 01:57:12.443829 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:57:12.443794 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-817fd7-764dbbc7f9-9fsbh" event={"ID":"f8de244f-5cf0-4543-9118-6684c2be86f0","Type":"ContainerStarted","Data":"73ad6962f718bda9b4dc6453a8ed6a38e66d6a98cd0e2813f05aae1443cf9f12"} Oct 30 01:57:12.444284 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:57:12.443854 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-817fd7-764dbbc7f9-9fsbh" event={"ID":"f8de244f-5cf0-4543-9118-6684c2be86f0","Type":"ContainerStarted","Data":"180620a2f20cd31938232d1cdf2c6db1d14aa4e6c72c3d0fdd1290bfd5ac68cd"} Oct 30 01:57:12.444284 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:57:12.444009 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-817fd7-764dbbc7f9-9fsbh" Oct 30 01:57:12.485895 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:57:12.485843 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-817fd7-764dbbc7f9-9fsbh" podStartSLOduration=1.007470448 podStartE2EDuration="1.485829785s" totalImagesPullingTime="478.359337ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:57:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:57:12.485373114 +0000 UTC m=+2356.063660451" watchObservedRunningTime="2026-10-30 01:57:12.485829785 +0000 UTC m=+2356.064117120" Oct 30 01:57:18.456878 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:57:18.456851 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-817fd7-764dbbc7f9-9fsbh" Oct 30 01:57:21.495036 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:57:21.494993 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:57:51.501306 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:57:51.501260 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:57:57.014639 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:57:57.014611 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2vsns_d332bbb4-c699-44e8-96ba-516d8ca03e95/ovn-acl-logging/0.log" Oct 30 01:57:57.015881 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:57:57.015857 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2vsns_d332bbb4-c699-44e8-96ba-516d8ca03e95/ovn-acl-logging/0.log" Oct 30 01:57:59.360203 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:57:59.360168 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:58:07.500051 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:07.500014 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-78572h7dk"] Oct 30 01:58:07.504733 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:07.504709 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-78572h7dk" Oct 30 01:58:07.524993 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:07.524966 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-78572h7dk"] Oct 30 01:58:07.532851 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:07.532825 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-093d23\"/\"agc-cred-1eeb94896bc0\"" Oct 30 01:58:07.533114 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:07.533092 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/057fc2ce-89fe-42ff-96d4-00d1edbe2aef-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-78572h7dk\" (UID: \"057fc2ce-89fe-42ff-96d4-00d1edbe2aef\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-78572h7dk" Oct 30 01:58:07.533170 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:07.533149 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/057fc2ce-89fe-42ff-96d4-00d1edbe2aef-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-78572h7dk\" (UID: \"057fc2ce-89fe-42ff-96d4-00d1edbe2aef\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-78572h7dk" Oct 30 01:58:07.533325 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:07.533313 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/057fc2ce-89fe-42ff-96d4-00d1edbe2aef-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-78572h7dk\" (UID: \"057fc2ce-89fe-42ff-96d4-00d1edbe2aef\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-78572h7dk" Oct 30 01:58:07.533363 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:07.533345 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47mtz\" (UniqueName: \"kubernetes.io/projected/057fc2ce-89fe-42ff-96d4-00d1edbe2aef-kube-api-access-47mtz\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-78572h7dk\" (UID: \"057fc2ce-89fe-42ff-96d4-00d1edbe2aef\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-78572h7dk" Oct 30 01:58:07.634145 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:07.634114 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/057fc2ce-89fe-42ff-96d4-00d1edbe2aef-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-78572h7dk\" (UID: \"057fc2ce-89fe-42ff-96d4-00d1edbe2aef\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-78572h7dk" Oct 30 01:58:07.634343 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:07.634170 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/057fc2ce-89fe-42ff-96d4-00d1edbe2aef-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-78572h7dk\" (UID: \"057fc2ce-89fe-42ff-96d4-00d1edbe2aef\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-78572h7dk" Oct 30 01:58:07.634469 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:07.634339 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/057fc2ce-89fe-42ff-96d4-00d1edbe2aef-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-78572h7dk\" (UID: \"057fc2ce-89fe-42ff-96d4-00d1edbe2aef\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-78572h7dk" Oct 30 01:58:07.634469 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:07.634386 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47mtz\" (UniqueName: \"kubernetes.io/projected/057fc2ce-89fe-42ff-96d4-00d1edbe2aef-kube-api-access-47mtz\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-78572h7dk\" (UID: \"057fc2ce-89fe-42ff-96d4-00d1edbe2aef\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-78572h7dk" Oct 30 01:58:07.635033 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:07.635001 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/057fc2ce-89fe-42ff-96d4-00d1edbe2aef-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-78572h7dk\" (UID: \"057fc2ce-89fe-42ff-96d4-00d1edbe2aef\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-78572h7dk" Oct 30 01:58:07.635163 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:07.635128 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/057fc2ce-89fe-42ff-96d4-00d1edbe2aef-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-78572h7dk\" (UID: \"057fc2ce-89fe-42ff-96d4-00d1edbe2aef\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-78572h7dk" Oct 30 01:58:07.637261 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:07.637238 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/057fc2ce-89fe-42ff-96d4-00d1edbe2aef-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-78572h7dk\" (UID: \"057fc2ce-89fe-42ff-96d4-00d1edbe2aef\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-78572h7dk" Oct 30 01:58:07.654874 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:07.654841 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47mtz\" (UniqueName: \"kubernetes.io/projected/057fc2ce-89fe-42ff-96d4-00d1edbe2aef-kube-api-access-47mtz\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-78572h7dk\" (UID: \"057fc2ce-89fe-42ff-96d4-00d1edbe2aef\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-78572h7dk" Oct 30 01:58:07.819079 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:07.819043 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-78572h7dk" Oct 30 01:58:08.125703 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:08.125673 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-78572h7dk"] Oct 30 01:58:08.128131 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:58:08.128104 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod057fc2ce_89fe_42ff_96d4_00d1edbe2aef.slice/crio-61a0ec5fa3a89aaee5601a45ed3db765fe7f126ef98f2fa3b0c2687381258020 WatchSource:0}: Error finding container 61a0ec5fa3a89aaee5601a45ed3db765fe7f126ef98f2fa3b0c2687381258020: Status 404 returned error can't find the container with id 61a0ec5fa3a89aaee5601a45ed3db765fe7f126ef98f2fa3b0c2687381258020 Oct 30 01:58:08.973497 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:08.973456 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-74d648zqm"] Oct 30 01:58:08.978266 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:08.978245 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-74d648zqm" Oct 30 01:58:08.996322 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:08.996298 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-74d648zqm"] Oct 30 01:58:09.004934 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:09.004904 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-cd8173\"/\"agc-cred-d60a14774030\"" Oct 30 01:58:09.005108 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:09.004932 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-cd8173\"/\"agc-cred-f70ae1064daa\"" Oct 30 01:58:09.005108 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:09.005090 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-cd8173\"/\"routing-overlay\"" Oct 30 01:58:09.005108 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:09.005096 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-cd8173\"/\"payload-processing-external-model-e2e--1a03a560-dockercfg-dwfpl\"" Oct 30 01:58:09.005286 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:09.005210 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-cd8173\"/\"payload-processing-external-model-plugins-e2e-extmod-b-cd8173\"" Oct 30 01:58:09.005286 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:09.005256 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-cd8173\"/\"kube-root-ca.crt\"" Oct 30 01:58:09.005429 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:09.005405 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-cd8173\"/\"openshift-service-ca.crt\"" Oct 30 01:58:09.055842 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:09.055808 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df18758c-b6f2-4adc-9fc5-19afe87a3d18-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-cd8173-74d648zqm\" (UID: \"df18758c-b6f2-4adc-9fc5-19afe87a3d18\") " pod="ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-74d648zqm" Oct 30 01:58:09.055997 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:09.055857 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df18758c-b6f2-4adc-9fc5-19afe87a3d18-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-cd8173-74d648zqm\" (UID: \"df18758c-b6f2-4adc-9fc5-19afe87a3d18\") " pod="ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-74d648zqm" Oct 30 01:58:09.056104 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:09.056078 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x959m\" (UniqueName: \"kubernetes.io/projected/df18758c-b6f2-4adc-9fc5-19afe87a3d18-kube-api-access-x959m\") pod \"payload-processing-external-model-e2e-extmod-b-cd8173-74d648zqm\" (UID: \"df18758c-b6f2-4adc-9fc5-19afe87a3d18\") " pod="ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-74d648zqm" Oct 30 01:58:09.056176 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:09.056164 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df18758c-b6f2-4adc-9fc5-19afe87a3d18-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-cd8173-74d648zqm\" (UID: \"df18758c-b6f2-4adc-9fc5-19afe87a3d18\") " pod="ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-74d648zqm" Oct 30 01:58:09.127786 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:09.127747 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-78572h7dk" event={"ID":"057fc2ce-89fe-42ff-96d4-00d1edbe2aef","Type":"ContainerStarted","Data":"34b179226a9b427ad2005fae96d0a977f93d0602c6e8d8e505e9818893f3fe0b"} Oct 30 01:58:09.127948 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:09.127813 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-78572h7dk" event={"ID":"057fc2ce-89fe-42ff-96d4-00d1edbe2aef","Type":"ContainerStarted","Data":"61a0ec5fa3a89aaee5601a45ed3db765fe7f126ef98f2fa3b0c2687381258020"} Oct 30 01:58:09.127948 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:09.127846 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-78572h7dk" Oct 30 01:58:09.157267 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:09.157234 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df18758c-b6f2-4adc-9fc5-19afe87a3d18-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-cd8173-74d648zqm\" (UID: \"df18758c-b6f2-4adc-9fc5-19afe87a3d18\") " pod="ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-74d648zqm" Oct 30 01:58:09.157440 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:09.157343 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df18758c-b6f2-4adc-9fc5-19afe87a3d18-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-cd8173-74d648zqm\" (UID: \"df18758c-b6f2-4adc-9fc5-19afe87a3d18\") " pod="ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-74d648zqm" Oct 30 01:58:09.157440 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:09.157402 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df18758c-b6f2-4adc-9fc5-19afe87a3d18-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-cd8173-74d648zqm\" (UID: \"df18758c-b6f2-4adc-9fc5-19afe87a3d18\") " pod="ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-74d648zqm" Oct 30 01:58:09.157759 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:09.157645 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x959m\" (UniqueName: \"kubernetes.io/projected/df18758c-b6f2-4adc-9fc5-19afe87a3d18-kube-api-access-x959m\") pod \"payload-processing-external-model-e2e-extmod-b-cd8173-74d648zqm\" (UID: \"df18758c-b6f2-4adc-9fc5-19afe87a3d18\") " pod="ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-74d648zqm" Oct 30 01:58:09.158116 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:09.158094 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df18758c-b6f2-4adc-9fc5-19afe87a3d18-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-cd8173-74d648zqm\" (UID: \"df18758c-b6f2-4adc-9fc5-19afe87a3d18\") " pod="ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-74d648zqm" Oct 30 01:58:09.158276 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:09.158251 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df18758c-b6f2-4adc-9fc5-19afe87a3d18-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-cd8173-74d648zqm\" (UID: \"df18758c-b6f2-4adc-9fc5-19afe87a3d18\") " pod="ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-74d648zqm" Oct 30 01:58:09.160185 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:09.160162 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df18758c-b6f2-4adc-9fc5-19afe87a3d18-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-cd8173-74d648zqm\" (UID: \"df18758c-b6f2-4adc-9fc5-19afe87a3d18\") " pod="ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-74d648zqm" Oct 30 01:58:09.168409 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:09.168362 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-78572h7dk" podStartSLOduration=2.168348682 podStartE2EDuration="2.168348682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:58:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:58:09.166159629 +0000 UTC m=+2412.744446964" watchObservedRunningTime="2026-10-30 01:58:09.168348682 +0000 UTC m=+2412.746636017" Oct 30 01:58:09.177842 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:09.177819 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x959m\" (UniqueName: \"kubernetes.io/projected/df18758c-b6f2-4adc-9fc5-19afe87a3d18-kube-api-access-x959m\") pod \"payload-processing-external-model-e2e-extmod-b-cd8173-74d648zqm\" (UID: \"df18758c-b6f2-4adc-9fc5-19afe87a3d18\") " pod="ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-74d648zqm" Oct 30 01:58:09.293758 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:09.293726 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-74d648zqm" Oct 30 01:58:09.600440 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:09.600411 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-74d648zqm"] Oct 30 01:58:09.602218 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:58:09.602187 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf18758c_b6f2_4adc_9fc5_19afe87a3d18.slice/crio-77855c59cec458c92c6ea971fedf89ab0adcf1d8019c8a6791769cf4f78941b2 WatchSource:0}: Error finding container 77855c59cec458c92c6ea971fedf89ab0adcf1d8019c8a6791769cf4f78941b2: Status 404 returned error can't find the container with id 77855c59cec458c92c6ea971fedf89ab0adcf1d8019c8a6791769cf4f78941b2 Oct 30 01:58:10.146933 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:10.146894 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-74d648zqm" event={"ID":"df18758c-b6f2-4adc-9fc5-19afe87a3d18","Type":"ContainerStarted","Data":"d1409f039eddf0c078e1b0be630966ed194f3cdbc88b9883b41951df654c83bd"} Oct 30 01:58:10.147427 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:10.147081 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-74d648zqm" Oct 30 01:58:10.147427 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:10.147122 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-74d648zqm" event={"ID":"df18758c-b6f2-4adc-9fc5-19afe87a3d18","Type":"ContainerStarted","Data":"77855c59cec458c92c6ea971fedf89ab0adcf1d8019c8a6791769cf4f78941b2"} Oct 30 01:58:10.188924 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:10.188874 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-74d648zqm" podStartSLOduration=2.188860546 podStartE2EDuration="2.188860546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:58:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:58:10.186218754 +0000 UTC m=+2413.764506091" watchObservedRunningTime="2026-10-30 01:58:10.188860546 +0000 UTC m=+2413.767147882" Oct 30 01:58:12.959679 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:12.959644 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-74d648zqm"] Oct 30 01:58:17.960824 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:17.960780 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-74d648zqm" podUID="df18758c-b6f2-4adc-9fc5-19afe87a3d18" containerName="payload-processing" containerID="cri-o://d1409f039eddf0c078e1b0be630966ed194f3cdbc88b9883b41951df654c83bd" gracePeriod=25 Oct 30 01:58:18.207402 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:18.207369 6221 generic.go:356] "Generic (PLEG): container finished" podID="df18758c-b6f2-4adc-9fc5-19afe87a3d18" containerID="d1409f039eddf0c078e1b0be630966ed194f3cdbc88b9883b41951df654c83bd" exitCode=0 Oct 30 01:58:18.207608 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:18.207442 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-74d648zqm" event={"ID":"df18758c-b6f2-4adc-9fc5-19afe87a3d18","Type":"ContainerDied","Data":"d1409f039eddf0c078e1b0be630966ed194f3cdbc88b9883b41951df654c83bd"} Oct 30 01:58:18.207608 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:18.207550 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-74d648zqm" event={"ID":"df18758c-b6f2-4adc-9fc5-19afe87a3d18","Type":"ContainerDied","Data":"77855c59cec458c92c6ea971fedf89ab0adcf1d8019c8a6791769cf4f78941b2"} Oct 30 01:58:18.207608 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:18.207576 6221 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77855c59cec458c92c6ea971fedf89ab0adcf1d8019c8a6791769cf4f78941b2" Oct 30 01:58:18.236579 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:18.236500 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-74d648zqm" Oct 30 01:58:18.352824 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:18.352795 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df18758c-b6f2-4adc-9fc5-19afe87a3d18-kube-api-access-x959m\" (UniqueName: \"kubernetes.io/projected/df18758c-b6f2-4adc-9fc5-19afe87a3d18-kube-api-access-x959m\") pod \"df18758c-b6f2-4adc-9fc5-19afe87a3d18\" (UID: \"df18758c-b6f2-4adc-9fc5-19afe87a3d18\") " Oct 30 01:58:18.352980 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:18.352864 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df18758c-b6f2-4adc-9fc5-19afe87a3d18-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df18758c-b6f2-4adc-9fc5-19afe87a3d18-routing-overlay-volume\") pod \"df18758c-b6f2-4adc-9fc5-19afe87a3d18\" (UID: \"df18758c-b6f2-4adc-9fc5-19afe87a3d18\") " Oct 30 01:58:18.353070 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:18.353054 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df18758c-b6f2-4adc-9fc5-19afe87a3d18-provider-credentials\" (UniqueName: \"kubernetes.io/projected/df18758c-b6f2-4adc-9fc5-19afe87a3d18-provider-credentials\") pod \"df18758c-b6f2-4adc-9fc5-19afe87a3d18\" (UID: \"df18758c-b6f2-4adc-9fc5-19afe87a3d18\") " Oct 30 01:58:18.353178 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:18.353160 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df18758c-b6f2-4adc-9fc5-19afe87a3d18-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df18758c-b6f2-4adc-9fc5-19afe87a3d18-plugins-config-volume\") pod \"df18758c-b6f2-4adc-9fc5-19afe87a3d18\" (UID: \"df18758c-b6f2-4adc-9fc5-19afe87a3d18\") " Oct 30 01:58:18.353270 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:18.353190 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df18758c-b6f2-4adc-9fc5-19afe87a3d18-routing-overlay-volume" pod "df18758c-b6f2-4adc-9fc5-19afe87a3d18" (UID: "df18758c-b6f2-4adc-9fc5-19afe87a3d18"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:58:18.353507 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:18.353480 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df18758c-b6f2-4adc-9fc5-19afe87a3d18-plugins-config-volume" pod "df18758c-b6f2-4adc-9fc5-19afe87a3d18" (UID: "df18758c-b6f2-4adc-9fc5-19afe87a3d18"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:58:18.353832 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:18.353813 6221 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/df18758c-b6f2-4adc-9fc5-19afe87a3d18-routing-overlay-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:58:18.353879 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:18.353834 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df18758c-b6f2-4adc-9fc5-19afe87a3d18-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:58:18.355175 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:18.355154 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df18758c-b6f2-4adc-9fc5-19afe87a3d18-kube-api-access-x959m" pod "df18758c-b6f2-4adc-9fc5-19afe87a3d18" (UID: "df18758c-b6f2-4adc-9fc5-19afe87a3d18"). InnerVolumeSpecName "kube-api-access-x959m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:58:18.355265 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:18.355201 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df18758c-b6f2-4adc-9fc5-19afe87a3d18-provider-credentials" pod "df18758c-b6f2-4adc-9fc5-19afe87a3d18" (UID: "df18758c-b6f2-4adc-9fc5-19afe87a3d18"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:58:18.455337 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:18.455304 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x959m\" (UniqueName: \"kubernetes.io/projected/df18758c-b6f2-4adc-9fc5-19afe87a3d18-kube-api-access-x959m\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:58:18.455337 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:18.455334 6221 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/df18758c-b6f2-4adc-9fc5-19afe87a3d18-provider-credentials\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:58:19.215163 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:19.215128 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-74d648zqm" Oct 30 01:58:19.268052 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:19.268018 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-74d648zqm"] Oct 30 01:58:19.280089 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:19.280060 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-cd8173/payload-processing-external-model-e2e-extmod-b-cd8173-74d648zqm"] Oct 30 01:58:20.149829 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:20.149797 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-78572h7dk" Oct 30 01:58:20.225948 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:20.225912 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76464j85z"] Oct 30 01:58:20.549032 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:20.548995 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df18758c-b6f2-4adc-9fc5-19afe87a3d18" path="/var/lib/kubelet/pods/df18758c-b6f2-4adc-9fc5-19afe87a3d18/volumes" Oct 30 01:58:21.505936 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:58:21.505892 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:58:25.227195 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:25.227154 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76464j85z" podUID="1db9df6e-e021-443b-8c97-139ccfc7c1cf" containerName="payload-processing" containerID="cri-o://b021cde7fd444ab7db6dd7857a3152ea1dc9fbba4c22beb96b5551669e13733a" gracePeriod=25 Oct 30 01:58:25.537868 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:25.537839 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76464j85z" Oct 30 01:58:25.580937 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:25.580904 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1db9df6e-e021-443b-8c97-139ccfc7c1cf-kube-api-access-cjz5c\" (UniqueName: \"kubernetes.io/projected/1db9df6e-e021-443b-8c97-139ccfc7c1cf-kube-api-access-cjz5c\") pod \"1db9df6e-e021-443b-8c97-139ccfc7c1cf\" (UID: \"1db9df6e-e021-443b-8c97-139ccfc7c1cf\") " Oct 30 01:58:25.581129 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:25.580963 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1db9df6e-e021-443b-8c97-139ccfc7c1cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1db9df6e-e021-443b-8c97-139ccfc7c1cf-plugins-config-volume\") pod \"1db9df6e-e021-443b-8c97-139ccfc7c1cf\" (UID: \"1db9df6e-e021-443b-8c97-139ccfc7c1cf\") " Oct 30 01:58:25.581129 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:25.581019 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1db9df6e-e021-443b-8c97-139ccfc7c1cf-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1db9df6e-e021-443b-8c97-139ccfc7c1cf-provider-credentials\") pod \"1db9df6e-e021-443b-8c97-139ccfc7c1cf\" (UID: \"1db9df6e-e021-443b-8c97-139ccfc7c1cf\") " Oct 30 01:58:25.581278 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:25.581250 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1db9df6e-e021-443b-8c97-139ccfc7c1cf-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1db9df6e-e021-443b-8c97-139ccfc7c1cf-routing-overlay-volume\") pod \"1db9df6e-e021-443b-8c97-139ccfc7c1cf\" (UID: \"1db9df6e-e021-443b-8c97-139ccfc7c1cf\") " Oct 30 01:58:25.581399 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:25.581368 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1db9df6e-e021-443b-8c97-139ccfc7c1cf-plugins-config-volume" pod "1db9df6e-e021-443b-8c97-139ccfc7c1cf" (UID: "1db9df6e-e021-443b-8c97-139ccfc7c1cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:58:25.581645 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:25.581620 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1db9df6e-e021-443b-8c97-139ccfc7c1cf-routing-overlay-volume" pod "1db9df6e-e021-443b-8c97-139ccfc7c1cf" (UID: "1db9df6e-e021-443b-8c97-139ccfc7c1cf"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:58:25.582504 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:25.582480 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1db9df6e-e021-443b-8c97-139ccfc7c1cf-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:58:25.582685 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:25.582506 6221 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1db9df6e-e021-443b-8c97-139ccfc7c1cf-routing-overlay-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:58:25.584147 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:25.584110 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1db9df6e-e021-443b-8c97-139ccfc7c1cf-kube-api-access-cjz5c" pod "1db9df6e-e021-443b-8c97-139ccfc7c1cf" (UID: "1db9df6e-e021-443b-8c97-139ccfc7c1cf"). InnerVolumeSpecName "kube-api-access-cjz5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:58:25.584471 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:25.584439 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1db9df6e-e021-443b-8c97-139ccfc7c1cf-provider-credentials" pod "1db9df6e-e021-443b-8c97-139ccfc7c1cf" (UID: "1db9df6e-e021-443b-8c97-139ccfc7c1cf"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:58:25.683503 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:25.683470 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjz5c\" (UniqueName: \"kubernetes.io/projected/1db9df6e-e021-443b-8c97-139ccfc7c1cf-kube-api-access-cjz5c\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:58:25.683503 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:25.683498 6221 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1db9df6e-e021-443b-8c97-139ccfc7c1cf-provider-credentials\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:58:26.270303 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:26.270266 6221 generic.go:356] "Generic (PLEG): container finished" podID="1db9df6e-e021-443b-8c97-139ccfc7c1cf" containerID="b021cde7fd444ab7db6dd7857a3152ea1dc9fbba4c22beb96b5551669e13733a" exitCode=0 Oct 30 01:58:26.270825 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:26.270308 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76464j85z" Oct 30 01:58:26.270825 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:26.270332 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76464j85z" event={"ID":"1db9df6e-e021-443b-8c97-139ccfc7c1cf","Type":"ContainerDied","Data":"b021cde7fd444ab7db6dd7857a3152ea1dc9fbba4c22beb96b5551669e13733a"} Oct 30 01:58:26.270825 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:26.270379 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76464j85z" event={"ID":"1db9df6e-e021-443b-8c97-139ccfc7c1cf","Type":"ContainerDied","Data":"5b9d9425569796e277575bd21f14ae04489763d623b6a414d15679a170364c1f"} Oct 30 01:58:26.270825 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:26.270409 6221 scope.go:122] "RemoveContainer" containerID="b021cde7fd444ab7db6dd7857a3152ea1dc9fbba4c22beb96b5551669e13733a" Oct 30 01:58:26.281971 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:26.281951 6221 scope.go:122] "RemoveContainer" containerID="b021cde7fd444ab7db6dd7857a3152ea1dc9fbba4c22beb96b5551669e13733a" Oct 30 01:58:26.282236 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:58:26.282218 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b021cde7fd444ab7db6dd7857a3152ea1dc9fbba4c22beb96b5551669e13733a\": container with ID starting with b021cde7fd444ab7db6dd7857a3152ea1dc9fbba4c22beb96b5551669e13733a not found: ID does not exist" containerID="b021cde7fd444ab7db6dd7857a3152ea1dc9fbba4c22beb96b5551669e13733a" Oct 30 01:58:26.282307 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:26.282245 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b021cde7fd444ab7db6dd7857a3152ea1dc9fbba4c22beb96b5551669e13733a"} err="failed to get container status \"b021cde7fd444ab7db6dd7857a3152ea1dc9fbba4c22beb96b5551669e13733a\": rpc error: code = NotFound desc = could not find container \"b021cde7fd444ab7db6dd7857a3152ea1dc9fbba4c22beb96b5551669e13733a\": container with ID starting with b021cde7fd444ab7db6dd7857a3152ea1dc9fbba4c22beb96b5551669e13733a not found: ID does not exist" Oct 30 01:58:26.340475 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:26.329137 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76464j85z"] Oct 30 01:58:26.372150 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:26.372117 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76464j85z"] Oct 30 01:58:26.658558 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:26.639681 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1db9df6e-e021-443b-8c97-139ccfc7c1cf" path="/var/lib/kubelet/pods/1db9df6e-e021-443b-8c97-139ccfc7c1cf/volumes" Oct 30 01:58:29.790059 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:29.790023 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-67754wlxf"] Oct 30 01:58:29.791148 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:29.791124 6221 cpu_manager.go:414] "containerMap: removing container" podUID="1db9df6e-e021-443b-8c97-139ccfc7c1cf" containerName="payload-processing" Oct 30 01:58:29.791216 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:29.791151 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1db9df6e-e021-443b-8c97-139ccfc7c1cf" containerName="payload-processing" Oct 30 01:58:29.791216 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:29.791200 6221 cpu_manager.go:414] "containerMap: removing container" podUID="df18758c-b6f2-4adc-9fc5-19afe87a3d18" containerName="payload-processing" Oct 30 01:58:29.791216 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:29.791208 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df18758c-b6f2-4adc-9fc5-19afe87a3d18" containerName="payload-processing" Oct 30 01:58:29.791388 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:29.791374 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="1db9df6e-e021-443b-8c97-139ccfc7c1cf" containerName="payload-processing" Oct 30 01:58:29.791438 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:29.791403 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="df18758c-b6f2-4adc-9fc5-19afe87a3d18" containerName="payload-processing" Oct 30 01:58:29.796635 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:29.796615 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-67754wlxf" Oct 30 01:58:29.812971 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:29.812941 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-67754wlxf"] Oct 30 01:58:29.820054 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:29.819984 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-76a19e\"/\"agc-cred-1786453aff48\"" Oct 30 01:58:29.820526 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:29.820505 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-76a19e\"/\"openshift-service-ca.crt\"" Oct 30 01:58:29.823275 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:29.823243 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-76a19e\"/\"routing-overlay\"" Oct 30 01:58:29.823726 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:29.823696 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-76a19e\"/\"kube-root-ca.crt\"" Oct 30 01:58:29.823894 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:29.823864 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-76a19e\"/\"payload-processing-external-model-e2e--70e20018-dockercfg-mldgh\"" Oct 30 01:58:29.824057 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:29.824039 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-76a19e\"/\"payload-processing-external-model-plugins-e2e-extmod-a-76a19e\"" Oct 30 01:58:29.867049 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:29.867015 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c446b5d7-3e82-4677-b71e-c17f7d1c4008-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-67754wlxf\" (UID: \"c446b5d7-3e82-4677-b71e-c17f7d1c4008\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-67754wlxf" Oct 30 01:58:29.867237 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:29.867105 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qlcv\" (UniqueName: \"kubernetes.io/projected/c446b5d7-3e82-4677-b71e-c17f7d1c4008-kube-api-access-2qlcv\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-67754wlxf\" (UID: \"c446b5d7-3e82-4677-b71e-c17f7d1c4008\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-67754wlxf" Oct 30 01:58:29.867379 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:29.867348 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c446b5d7-3e82-4677-b71e-c17f7d1c4008-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-67754wlxf\" (UID: \"c446b5d7-3e82-4677-b71e-c17f7d1c4008\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-67754wlxf" Oct 30 01:58:29.867490 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:29.867471 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c446b5d7-3e82-4677-b71e-c17f7d1c4008-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-67754wlxf\" (UID: \"c446b5d7-3e82-4677-b71e-c17f7d1c4008\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-67754wlxf" Oct 30 01:58:29.977554 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:29.977471 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c446b5d7-3e82-4677-b71e-c17f7d1c4008-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-67754wlxf\" (UID: \"c446b5d7-3e82-4677-b71e-c17f7d1c4008\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-67754wlxf" Oct 30 01:58:29.977771 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:29.977600 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c446b5d7-3e82-4677-b71e-c17f7d1c4008-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-67754wlxf\" (UID: \"c446b5d7-3e82-4677-b71e-c17f7d1c4008\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-67754wlxf" Oct 30 01:58:29.977771 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:29.977691 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c446b5d7-3e82-4677-b71e-c17f7d1c4008-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-67754wlxf\" (UID: \"c446b5d7-3e82-4677-b71e-c17f7d1c4008\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-67754wlxf" Oct 30 01:58:29.977923 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:29.977804 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qlcv\" (UniqueName: \"kubernetes.io/projected/c446b5d7-3e82-4677-b71e-c17f7d1c4008-kube-api-access-2qlcv\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-67754wlxf\" (UID: \"c446b5d7-3e82-4677-b71e-c17f7d1c4008\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-67754wlxf" Oct 30 01:58:29.981551 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:29.979059 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c446b5d7-3e82-4677-b71e-c17f7d1c4008-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-67754wlxf\" (UID: \"c446b5d7-3e82-4677-b71e-c17f7d1c4008\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-67754wlxf" Oct 30 01:58:30.001553 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:30.001042 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c446b5d7-3e82-4677-b71e-c17f7d1c4008-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-67754wlxf\" (UID: \"c446b5d7-3e82-4677-b71e-c17f7d1c4008\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-67754wlxf" Oct 30 01:58:30.005781 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:30.004145 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c446b5d7-3e82-4677-b71e-c17f7d1c4008-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-67754wlxf\" (UID: \"c446b5d7-3e82-4677-b71e-c17f7d1c4008\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-67754wlxf" Oct 30 01:58:30.047136 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:30.042074 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qlcv\" (UniqueName: \"kubernetes.io/projected/c446b5d7-3e82-4677-b71e-c17f7d1c4008-kube-api-access-2qlcv\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-67754wlxf\" (UID: \"c446b5d7-3e82-4677-b71e-c17f7d1c4008\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-67754wlxf" Oct 30 01:58:30.113648 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:30.113603 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-67754wlxf" Oct 30 01:58:30.419742 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:30.419710 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-67754wlxf"] Oct 30 01:58:30.422574 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:58:30.422543 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc446b5d7_3e82_4677_b71e_c17f7d1c4008.slice/crio-02a5e1910963aea221c59c0f468ef3b3c62bf0dd519f7f5ad0b6c3842098e3de WatchSource:0}: Error finding container 02a5e1910963aea221c59c0f468ef3b3c62bf0dd519f7f5ad0b6c3842098e3de: Status 404 returned error can't find the container with id 02a5e1910963aea221c59c0f468ef3b3c62bf0dd519f7f5ad0b6c3842098e3de Oct 30 01:58:31.312578 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:31.312513 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-67754wlxf" event={"ID":"c446b5d7-3e82-4677-b71e-c17f7d1c4008","Type":"ContainerStarted","Data":"b9d2bed80acc8da10142b45328105248d7587d33faf04e372a7bb30cd95d5e35"} Oct 30 01:58:31.313010 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:31.312609 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-67754wlxf" Oct 30 01:58:31.313010 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:31.312636 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-67754wlxf" event={"ID":"c446b5d7-3e82-4677-b71e-c17f7d1c4008","Type":"ContainerStarted","Data":"02a5e1910963aea221c59c0f468ef3b3c62bf0dd519f7f5ad0b6c3842098e3de"} Oct 30 01:58:31.357469 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:31.357412 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-67754wlxf" podStartSLOduration=2.357398088 podStartE2EDuration="2.357398088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:58:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:58:31.353833902 +0000 UTC m=+2434.932121264" watchObservedRunningTime="2026-10-30 01:58:31.357398088 +0000 UTC m=+2434.935685424" Oct 30 01:58:42.324968 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:42.324883 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-67754wlxf" Oct 30 01:58:43.088503 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:43.088468 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-67754wlxf"] Oct 30 01:58:48.088802 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:48.088758 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-67754wlxf" podUID="c446b5d7-3e82-4677-b71e-c17f7d1c4008" containerName="payload-processing" containerID="cri-o://b9d2bed80acc8da10142b45328105248d7587d33faf04e372a7bb30cd95d5e35" gracePeriod=25 Oct 30 01:58:48.373033 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:48.373004 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-67754wlxf" Oct 30 01:58:48.453074 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:48.452149 6221 generic.go:356] "Generic (PLEG): container finished" podID="c446b5d7-3e82-4677-b71e-c17f7d1c4008" containerID="b9d2bed80acc8da10142b45328105248d7587d33faf04e372a7bb30cd95d5e35" exitCode=0 Oct 30 01:58:48.453074 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:48.452343 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-67754wlxf" event={"ID":"c446b5d7-3e82-4677-b71e-c17f7d1c4008","Type":"ContainerDied","Data":"b9d2bed80acc8da10142b45328105248d7587d33faf04e372a7bb30cd95d5e35"} Oct 30 01:58:48.453074 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:48.452395 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-67754wlxf" event={"ID":"c446b5d7-3e82-4677-b71e-c17f7d1c4008","Type":"ContainerDied","Data":"02a5e1910963aea221c59c0f468ef3b3c62bf0dd519f7f5ad0b6c3842098e3de"} Oct 30 01:58:48.453074 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:48.452424 6221 scope.go:122] "RemoveContainer" containerID="b9d2bed80acc8da10142b45328105248d7587d33faf04e372a7bb30cd95d5e35" Oct 30 01:58:48.453074 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:48.452626 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-67754wlxf" Oct 30 01:58:48.465102 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:48.465084 6221 scope.go:122] "RemoveContainer" containerID="b9d2bed80acc8da10142b45328105248d7587d33faf04e372a7bb30cd95d5e35" Oct 30 01:58:48.465374 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:58:48.465351 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9d2bed80acc8da10142b45328105248d7587d33faf04e372a7bb30cd95d5e35\": container with ID starting with b9d2bed80acc8da10142b45328105248d7587d33faf04e372a7bb30cd95d5e35 not found: ID does not exist" containerID="b9d2bed80acc8da10142b45328105248d7587d33faf04e372a7bb30cd95d5e35" Oct 30 01:58:48.465440 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:48.465383 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9d2bed80acc8da10142b45328105248d7587d33faf04e372a7bb30cd95d5e35"} err="failed to get container status \"b9d2bed80acc8da10142b45328105248d7587d33faf04e372a7bb30cd95d5e35\": rpc error: code = NotFound desc = could not find container \"b9d2bed80acc8da10142b45328105248d7587d33faf04e372a7bb30cd95d5e35\": container with ID starting with b9d2bed80acc8da10142b45328105248d7587d33faf04e372a7bb30cd95d5e35 not found: ID does not exist" Oct 30 01:58:48.471945 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:48.471923 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c446b5d7-3e82-4677-b71e-c17f7d1c4008-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c446b5d7-3e82-4677-b71e-c17f7d1c4008-plugins-config-volume\") pod \"c446b5d7-3e82-4677-b71e-c17f7d1c4008\" (UID: \"c446b5d7-3e82-4677-b71e-c17f7d1c4008\") " Oct 30 01:58:48.472040 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:48.472034 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c446b5d7-3e82-4677-b71e-c17f7d1c4008-kube-api-access-2qlcv\" (UniqueName: \"kubernetes.io/projected/c446b5d7-3e82-4677-b71e-c17f7d1c4008-kube-api-access-2qlcv\") pod \"c446b5d7-3e82-4677-b71e-c17f7d1c4008\" (UID: \"c446b5d7-3e82-4677-b71e-c17f7d1c4008\") " Oct 30 01:58:48.472089 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:48.472070 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c446b5d7-3e82-4677-b71e-c17f7d1c4008-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c446b5d7-3e82-4677-b71e-c17f7d1c4008-provider-credentials\") pod \"c446b5d7-3e82-4677-b71e-c17f7d1c4008\" (UID: \"c446b5d7-3e82-4677-b71e-c17f7d1c4008\") " Oct 30 01:58:48.472133 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:48.472117 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c446b5d7-3e82-4677-b71e-c17f7d1c4008-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c446b5d7-3e82-4677-b71e-c17f7d1c4008-routing-overlay-volume\") pod \"c446b5d7-3e82-4677-b71e-c17f7d1c4008\" (UID: \"c446b5d7-3e82-4677-b71e-c17f7d1c4008\") " Oct 30 01:58:48.472349 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:48.472318 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c446b5d7-3e82-4677-b71e-c17f7d1c4008-plugins-config-volume" pod "c446b5d7-3e82-4677-b71e-c17f7d1c4008" (UID: "c446b5d7-3e82-4677-b71e-c17f7d1c4008"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:58:48.472550 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:48.472515 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c446b5d7-3e82-4677-b71e-c17f7d1c4008-routing-overlay-volume" pod "c446b5d7-3e82-4677-b71e-c17f7d1c4008" (UID: "c446b5d7-3e82-4677-b71e-c17f7d1c4008"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:58:48.472794 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:48.472776 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c446b5d7-3e82-4677-b71e-c17f7d1c4008-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:58:48.472863 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:48.472796 6221 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c446b5d7-3e82-4677-b71e-c17f7d1c4008-routing-overlay-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:58:48.474359 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:48.474340 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c446b5d7-3e82-4677-b71e-c17f7d1c4008-kube-api-access-2qlcv" pod "c446b5d7-3e82-4677-b71e-c17f7d1c4008" (UID: "c446b5d7-3e82-4677-b71e-c17f7d1c4008"). InnerVolumeSpecName "kube-api-access-2qlcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:58:48.474442 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:48.474366 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c446b5d7-3e82-4677-b71e-c17f7d1c4008-provider-credentials" pod "c446b5d7-3e82-4677-b71e-c17f7d1c4008" (UID: "c446b5d7-3e82-4677-b71e-c17f7d1c4008"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:58:48.575412 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:48.575372 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qlcv\" (UniqueName: \"kubernetes.io/projected/c446b5d7-3e82-4677-b71e-c17f7d1c4008-kube-api-access-2qlcv\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:58:48.575412 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:48.575415 6221 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c446b5d7-3e82-4677-b71e-c17f7d1c4008-provider-credentials\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:58:48.803101 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:48.803071 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-67754wlxf"] Oct 30 01:58:48.807280 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:48.807255 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-67754wlxf"] Oct 30 01:58:50.549069 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:50.549031 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c446b5d7-3e82-4677-b71e-c17f7d1c4008" path="/var/lib/kubelet/pods/c446b5d7-3e82-4677-b71e-c17f7d1c4008/volumes" Oct 30 01:58:51.498464 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:58:51.498420 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:58:52.864194 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:52.864158 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:58:56.973563 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:56.973499 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm"] Oct 30 01:58:56.974303 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:56.974282 6221 cpu_manager.go:414] "containerMap: removing container" podUID="c446b5d7-3e82-4677-b71e-c17f7d1c4008" containerName="payload-processing" Oct 30 01:58:56.974303 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:56.974302 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c446b5d7-3e82-4677-b71e-c17f7d1c4008" containerName="payload-processing" Oct 30 01:58:56.974562 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:56.974432 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="c446b5d7-3e82-4677-b71e-c17f7d1c4008" containerName="payload-processing" Oct 30 01:58:56.977901 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:56.977882 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm" Oct 30 01:58:56.996123 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:56.996089 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm"] Oct 30 01:58:57.004159 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:57.004135 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-093d23\"/\"agc-cred-723128d71ad3\"" Oct 30 01:58:57.057563 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:57.057487 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfx55\" (UniqueName: \"kubernetes.io/projected/07b7fc55-3b8b-4019-aae0-88231682d45e-kube-api-access-gfx55\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm\" (UID: \"07b7fc55-3b8b-4019-aae0-88231682d45e\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm" Oct 30 01:58:57.057757 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:57.057668 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07b7fc55-3b8b-4019-aae0-88231682d45e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm\" (UID: \"07b7fc55-3b8b-4019-aae0-88231682d45e\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm" Oct 30 01:58:57.057815 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:57.057795 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/07b7fc55-3b8b-4019-aae0-88231682d45e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm\" (UID: \"07b7fc55-3b8b-4019-aae0-88231682d45e\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm" Oct 30 01:58:57.057867 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:57.057854 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07b7fc55-3b8b-4019-aae0-88231682d45e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm\" (UID: \"07b7fc55-3b8b-4019-aae0-88231682d45e\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm" Oct 30 01:58:57.158909 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:57.158874 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfx55\" (UniqueName: \"kubernetes.io/projected/07b7fc55-3b8b-4019-aae0-88231682d45e-kube-api-access-gfx55\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm\" (UID: \"07b7fc55-3b8b-4019-aae0-88231682d45e\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm" Oct 30 01:58:57.159087 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:57.158955 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07b7fc55-3b8b-4019-aae0-88231682d45e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm\" (UID: \"07b7fc55-3b8b-4019-aae0-88231682d45e\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm" Oct 30 01:58:57.159087 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:57.159015 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/07b7fc55-3b8b-4019-aae0-88231682d45e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm\" (UID: \"07b7fc55-3b8b-4019-aae0-88231682d45e\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm" Oct 30 01:58:57.159087 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:57.159057 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07b7fc55-3b8b-4019-aae0-88231682d45e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm\" (UID: \"07b7fc55-3b8b-4019-aae0-88231682d45e\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm" Oct 30 01:58:57.159679 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:57.159652 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07b7fc55-3b8b-4019-aae0-88231682d45e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm\" (UID: \"07b7fc55-3b8b-4019-aae0-88231682d45e\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm" Oct 30 01:58:57.159828 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:57.159809 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07b7fc55-3b8b-4019-aae0-88231682d45e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm\" (UID: \"07b7fc55-3b8b-4019-aae0-88231682d45e\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm" Oct 30 01:58:57.161671 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:57.161648 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/07b7fc55-3b8b-4019-aae0-88231682d45e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm\" (UID: \"07b7fc55-3b8b-4019-aae0-88231682d45e\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm" Oct 30 01:58:57.179582 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:57.179557 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfx55\" (UniqueName: \"kubernetes.io/projected/07b7fc55-3b8b-4019-aae0-88231682d45e-kube-api-access-gfx55\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm\" (UID: \"07b7fc55-3b8b-4019-aae0-88231682d45e\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm" Oct 30 01:58:57.293443 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:57.293409 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm" Oct 30 01:58:57.598636 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:57.598607 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm"] Oct 30 01:58:57.600589 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:58:57.600561 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07b7fc55_3b8b_4019_aae0_88231682d45e.slice/crio-abbba351d598363ae6ca391b0592334a85e4569335dbae30d849cd9d4d18e26e WatchSource:0}: Error finding container abbba351d598363ae6ca391b0592334a85e4569335dbae30d849cd9d4d18e26e: Status 404 returned error can't find the container with id abbba351d598363ae6ca391b0592334a85e4569335dbae30d849cd9d4d18e26e Oct 30 01:58:58.612813 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:58.612770 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm" event={"ID":"07b7fc55-3b8b-4019-aae0-88231682d45e","Type":"ContainerStarted","Data":"4e16b7ee26d0985febad68a2dccd8fcd9e71783ee0f46fc55897f64ef1cd2426"} Oct 30 01:58:58.613229 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:58.612836 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm" event={"ID":"07b7fc55-3b8b-4019-aae0-88231682d45e","Type":"ContainerStarted","Data":"abbba351d598363ae6ca391b0592334a85e4569335dbae30d849cd9d4d18e26e"} Oct 30 01:58:58.613229 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:58.612868 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm" Oct 30 01:58:58.656279 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:58.656225 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm" podStartSLOduration=2.656210621 podStartE2EDuration="2.656210621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:58:58.65329239 +0000 UTC m=+2462.231579726" watchObservedRunningTime="2026-10-30 01:58:58.656210621 +0000 UTC m=+2462.234497957" Oct 30 01:58:59.494960 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:59.494927 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67v6z8r"] Oct 30 01:58:59.500862 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:59.500836 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67v6z8r" Oct 30 01:58:59.523716 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:59.523684 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67v6z8r"] Oct 30 01:58:59.534009 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:59.533968 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-92f0cc\"/\"openshift-service-ca.crt\"" Oct 30 01:58:59.535306 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:59.535280 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-92f0cc\"/\"agc-cred-2bf58c377925\"" Oct 30 01:58:59.535650 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:59.535587 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-92f0cc\"/\"routing-overlay\"" Oct 30 01:58:59.535934 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:59.535911 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-92f0cc\"/\"payload-processing-external-model-plugins-e2e-extmod-a-92f0cc\"" Oct 30 01:58:59.536079 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:59.535990 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-92f0cc\"/\"agc-cred-ce62c5464e20\"" Oct 30 01:58:59.536171 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:59.536112 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-92f0cc\"/\"payload-processing-external-model-e2e--c7e40432-dockercfg-fffn4\"" Oct 30 01:58:59.536306 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:59.536174 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-92f0cc\"/\"kube-root-ca.crt\"" Oct 30 01:58:59.638393 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:59.638351 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cb0d5c5a-987f-4b86-b072-3147e343d83d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-6c67v6z8r\" (UID: \"cb0d5c5a-987f-4b86-b072-3147e343d83d\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67v6z8r" Oct 30 01:58:59.638849 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:59.638414 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cb0d5c5a-987f-4b86-b072-3147e343d83d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-6c67v6z8r\" (UID: \"cb0d5c5a-987f-4b86-b072-3147e343d83d\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67v6z8r" Oct 30 01:58:59.638849 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:59.638600 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb0d5c5a-987f-4b86-b072-3147e343d83d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-6c67v6z8r\" (UID: \"cb0d5c5a-987f-4b86-b072-3147e343d83d\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67v6z8r" Oct 30 01:58:59.638849 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:59.638731 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69mz5\" (UniqueName: \"kubernetes.io/projected/cb0d5c5a-987f-4b86-b072-3147e343d83d-kube-api-access-69mz5\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-6c67v6z8r\" (UID: \"cb0d5c5a-987f-4b86-b072-3147e343d83d\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67v6z8r" Oct 30 01:58:59.739847 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:59.739809 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cb0d5c5a-987f-4b86-b072-3147e343d83d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-6c67v6z8r\" (UID: \"cb0d5c5a-987f-4b86-b072-3147e343d83d\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67v6z8r" Oct 30 01:58:59.740031 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:59.739863 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cb0d5c5a-987f-4b86-b072-3147e343d83d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-6c67v6z8r\" (UID: \"cb0d5c5a-987f-4b86-b072-3147e343d83d\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67v6z8r" Oct 30 01:58:59.740031 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:59.740008 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb0d5c5a-987f-4b86-b072-3147e343d83d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-6c67v6z8r\" (UID: \"cb0d5c5a-987f-4b86-b072-3147e343d83d\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67v6z8r" Oct 30 01:58:59.740166 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:59.740131 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69mz5\" (UniqueName: \"kubernetes.io/projected/cb0d5c5a-987f-4b86-b072-3147e343d83d-kube-api-access-69mz5\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-6c67v6z8r\" (UID: \"cb0d5c5a-987f-4b86-b072-3147e343d83d\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67v6z8r" Oct 30 01:58:59.740651 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:59.740619 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cb0d5c5a-987f-4b86-b072-3147e343d83d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-6c67v6z8r\" (UID: \"cb0d5c5a-987f-4b86-b072-3147e343d83d\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67v6z8r" Oct 30 01:58:59.740827 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:59.740807 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb0d5c5a-987f-4b86-b072-3147e343d83d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-6c67v6z8r\" (UID: \"cb0d5c5a-987f-4b86-b072-3147e343d83d\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67v6z8r" Oct 30 01:58:59.742590 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:59.742571 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cb0d5c5a-987f-4b86-b072-3147e343d83d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-6c67v6z8r\" (UID: \"cb0d5c5a-987f-4b86-b072-3147e343d83d\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67v6z8r" Oct 30 01:58:59.759693 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:59.759628 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69mz5\" (UniqueName: \"kubernetes.io/projected/cb0d5c5a-987f-4b86-b072-3147e343d83d-kube-api-access-69mz5\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-6c67v6z8r\" (UID: \"cb0d5c5a-987f-4b86-b072-3147e343d83d\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67v6z8r" Oct 30 01:58:59.819839 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:58:59.819806 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67v6z8r" Oct 30 01:59:00.131865 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:00.131838 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67v6z8r"] Oct 30 01:59:00.134146 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:59:00.134116 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb0d5c5a_987f_4b86_b072_3147e343d83d.slice/crio-837429fb9380948db1b3032942419e9648e9b2dcd2b1d9250a01cb5caae980f7 WatchSource:0}: Error finding container 837429fb9380948db1b3032942419e9648e9b2dcd2b1d9250a01cb5caae980f7: Status 404 returned error can't find the container with id 837429fb9380948db1b3032942419e9648e9b2dcd2b1d9250a01cb5caae980f7 Oct 30 01:59:00.631049 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:00.631010 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67v6z8r" event={"ID":"cb0d5c5a-987f-4b86-b072-3147e343d83d","Type":"ContainerStarted","Data":"3dcce1046d1a547d5fd962e56e1d3a104532ff4ad7eb4cb14978ac52736486d5"} Oct 30 01:59:00.631262 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:00.631069 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67v6z8r" event={"ID":"cb0d5c5a-987f-4b86-b072-3147e343d83d","Type":"ContainerStarted","Data":"837429fb9380948db1b3032942419e9648e9b2dcd2b1d9250a01cb5caae980f7"} Oct 30 01:59:00.631262 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:00.631119 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67v6z8r" Oct 30 01:59:00.672756 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:00.672700 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67v6z8r" podStartSLOduration=1.67268516 podStartE2EDuration="1.67268516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:59:00.67088091 +0000 UTC m=+2464.249168246" watchObservedRunningTime="2026-10-30 01:59:00.67268516 +0000 UTC m=+2464.250972513" Oct 30 01:59:05.434744 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:05.434710 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67v6z8r"] Oct 30 01:59:05.437503 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:05.437467 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67v6z8r" Oct 30 01:59:09.622825 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:09.622798 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm" Oct 30 01:59:09.693517 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:09.693459 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-78572h7dk"] Oct 30 01:59:10.435815 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:10.435767 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67v6z8r" podUID="cb0d5c5a-987f-4b86-b072-3147e343d83d" containerName="payload-processing" containerID="cri-o://3dcce1046d1a547d5fd962e56e1d3a104532ff4ad7eb4cb14978ac52736486d5" gracePeriod=25 Oct 30 01:59:10.710644 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:10.710618 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67v6z8r" Oct 30 01:59:10.804709 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:10.804670 6221 generic.go:356] "Generic (PLEG): container finished" podID="cb0d5c5a-987f-4b86-b072-3147e343d83d" containerID="3dcce1046d1a547d5fd962e56e1d3a104532ff4ad7eb4cb14978ac52736486d5" exitCode=0 Oct 30 01:59:10.804946 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:10.804925 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67v6z8r" event={"ID":"cb0d5c5a-987f-4b86-b072-3147e343d83d","Type":"ContainerDied","Data":"3dcce1046d1a547d5fd962e56e1d3a104532ff4ad7eb4cb14978ac52736486d5"} Oct 30 01:59:10.805038 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:10.805022 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67v6z8r" event={"ID":"cb0d5c5a-987f-4b86-b072-3147e343d83d","Type":"ContainerDied","Data":"837429fb9380948db1b3032942419e9648e9b2dcd2b1d9250a01cb5caae980f7"} Oct 30 01:59:10.805088 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:10.805060 6221 scope.go:122] "RemoveContainer" containerID="3dcce1046d1a547d5fd962e56e1d3a104532ff4ad7eb4cb14978ac52736486d5" Oct 30 01:59:10.805245 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:10.805230 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67v6z8r" Oct 30 01:59:10.815988 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:10.815964 6221 scope.go:122] "RemoveContainer" containerID="3dcce1046d1a547d5fd962e56e1d3a104532ff4ad7eb4cb14978ac52736486d5" Oct 30 01:59:10.816346 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:59:10.816320 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dcce1046d1a547d5fd962e56e1d3a104532ff4ad7eb4cb14978ac52736486d5\": container with ID starting with 3dcce1046d1a547d5fd962e56e1d3a104532ff4ad7eb4cb14978ac52736486d5 not found: ID does not exist" containerID="3dcce1046d1a547d5fd962e56e1d3a104532ff4ad7eb4cb14978ac52736486d5" Oct 30 01:59:10.816452 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:10.816359 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dcce1046d1a547d5fd962e56e1d3a104532ff4ad7eb4cb14978ac52736486d5"} err="failed to get container status \"3dcce1046d1a547d5fd962e56e1d3a104532ff4ad7eb4cb14978ac52736486d5\": rpc error: code = NotFound desc = could not find container \"3dcce1046d1a547d5fd962e56e1d3a104532ff4ad7eb4cb14978ac52736486d5\": container with ID starting with 3dcce1046d1a547d5fd962e56e1d3a104532ff4ad7eb4cb14978ac52736486d5 not found: ID does not exist" Oct 30 01:59:10.842267 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:10.842236 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb0d5c5a-987f-4b86-b072-3147e343d83d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/cb0d5c5a-987f-4b86-b072-3147e343d83d-provider-credentials\") pod \"cb0d5c5a-987f-4b86-b072-3147e343d83d\" (UID: \"cb0d5c5a-987f-4b86-b072-3147e343d83d\") " Oct 30 01:59:10.842417 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:10.842408 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb0d5c5a-987f-4b86-b072-3147e343d83d-kube-api-access-69mz5\" (UniqueName: \"kubernetes.io/projected/cb0d5c5a-987f-4b86-b072-3147e343d83d-kube-api-access-69mz5\") pod \"cb0d5c5a-987f-4b86-b072-3147e343d83d\" (UID: \"cb0d5c5a-987f-4b86-b072-3147e343d83d\") " Oct 30 01:59:10.842473 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:10.842431 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb0d5c5a-987f-4b86-b072-3147e343d83d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb0d5c5a-987f-4b86-b072-3147e343d83d-plugins-config-volume\") pod \"cb0d5c5a-987f-4b86-b072-3147e343d83d\" (UID: \"cb0d5c5a-987f-4b86-b072-3147e343d83d\") " Oct 30 01:59:10.842513 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:10.842486 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb0d5c5a-987f-4b86-b072-3147e343d83d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cb0d5c5a-987f-4b86-b072-3147e343d83d-routing-overlay-volume\") pod \"cb0d5c5a-987f-4b86-b072-3147e343d83d\" (UID: \"cb0d5c5a-987f-4b86-b072-3147e343d83d\") " Oct 30 01:59:10.842869 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:10.842844 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb0d5c5a-987f-4b86-b072-3147e343d83d-routing-overlay-volume" pod "cb0d5c5a-987f-4b86-b072-3147e343d83d" (UID: "cb0d5c5a-987f-4b86-b072-3147e343d83d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:59:10.842999 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:10.842900 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb0d5c5a-987f-4b86-b072-3147e343d83d-plugins-config-volume" pod "cb0d5c5a-987f-4b86-b072-3147e343d83d" (UID: "cb0d5c5a-987f-4b86-b072-3147e343d83d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:59:10.843062 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:10.842996 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb0d5c5a-987f-4b86-b072-3147e343d83d-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:59:10.843062 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:10.843013 6221 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cb0d5c5a-987f-4b86-b072-3147e343d83d-routing-overlay-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:59:10.844683 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:10.844660 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb0d5c5a-987f-4b86-b072-3147e343d83d-provider-credentials" pod "cb0d5c5a-987f-4b86-b072-3147e343d83d" (UID: "cb0d5c5a-987f-4b86-b072-3147e343d83d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:59:10.844782 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:10.844711 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb0d5c5a-987f-4b86-b072-3147e343d83d-kube-api-access-69mz5" pod "cb0d5c5a-987f-4b86-b072-3147e343d83d" (UID: "cb0d5c5a-987f-4b86-b072-3147e343d83d"). InnerVolumeSpecName "kube-api-access-69mz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:59:10.944459 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:10.944364 6221 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cb0d5c5a-987f-4b86-b072-3147e343d83d-provider-credentials\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:59:10.944459 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:10.944403 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69mz5\" (UniqueName: \"kubernetes.io/projected/cb0d5c5a-987f-4b86-b072-3147e343d83d-kube-api-access-69mz5\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:59:11.152935 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:11.152903 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67v6z8r"] Oct 30 01:59:11.164582 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:11.164553 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-6c67v6z8r"] Oct 30 01:59:12.548907 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:12.548875 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb0d5c5a-987f-4b86-b072-3147e343d83d" path="/var/lib/kubelet/pods/cb0d5c5a-987f-4b86-b072-3147e343d83d/volumes" Oct 30 01:59:14.693902 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:14.693835 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-78572h7dk" podUID="057fc2ce-89fe-42ff-96d4-00d1edbe2aef" containerName="payload-processing" containerID="cri-o://34b179226a9b427ad2005fae96d0a977f93d0602c6e8d8e505e9818893f3fe0b" gracePeriod=25 Oct 30 01:59:14.897280 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:14.897246 6221 generic.go:356] "Generic (PLEG): container finished" podID="057fc2ce-89fe-42ff-96d4-00d1edbe2aef" containerID="34b179226a9b427ad2005fae96d0a977f93d0602c6e8d8e505e9818893f3fe0b" exitCode=0 Oct 30 01:59:14.897482 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:14.897315 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-78572h7dk" event={"ID":"057fc2ce-89fe-42ff-96d4-00d1edbe2aef","Type":"ContainerDied","Data":"34b179226a9b427ad2005fae96d0a977f93d0602c6e8d8e505e9818893f3fe0b"} Oct 30 01:59:15.103036 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:15.103007 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-78572h7dk" Oct 30 01:59:15.223627 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:15.223596 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/057fc2ce-89fe-42ff-96d4-00d1edbe2aef-kube-api-access-47mtz\" (UniqueName: \"kubernetes.io/projected/057fc2ce-89fe-42ff-96d4-00d1edbe2aef-kube-api-access-47mtz\") pod \"057fc2ce-89fe-42ff-96d4-00d1edbe2aef\" (UID: \"057fc2ce-89fe-42ff-96d4-00d1edbe2aef\") " Oct 30 01:59:15.223810 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:15.223694 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/057fc2ce-89fe-42ff-96d4-00d1edbe2aef-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/057fc2ce-89fe-42ff-96d4-00d1edbe2aef-routing-overlay-volume\") pod \"057fc2ce-89fe-42ff-96d4-00d1edbe2aef\" (UID: \"057fc2ce-89fe-42ff-96d4-00d1edbe2aef\") " Oct 30 01:59:15.223810 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:15.223720 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/057fc2ce-89fe-42ff-96d4-00d1edbe2aef-provider-credentials\" (UniqueName: \"kubernetes.io/projected/057fc2ce-89fe-42ff-96d4-00d1edbe2aef-provider-credentials\") pod \"057fc2ce-89fe-42ff-96d4-00d1edbe2aef\" (UID: \"057fc2ce-89fe-42ff-96d4-00d1edbe2aef\") " Oct 30 01:59:15.224016 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:15.223855 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/057fc2ce-89fe-42ff-96d4-00d1edbe2aef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/057fc2ce-89fe-42ff-96d4-00d1edbe2aef-plugins-config-volume\") pod \"057fc2ce-89fe-42ff-96d4-00d1edbe2aef\" (UID: \"057fc2ce-89fe-42ff-96d4-00d1edbe2aef\") " Oct 30 01:59:15.224154 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:15.224129 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/057fc2ce-89fe-42ff-96d4-00d1edbe2aef-routing-overlay-volume" pod "057fc2ce-89fe-42ff-96d4-00d1edbe2aef" (UID: "057fc2ce-89fe-42ff-96d4-00d1edbe2aef"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:59:15.224236 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:15.224216 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/057fc2ce-89fe-42ff-96d4-00d1edbe2aef-plugins-config-volume" pod "057fc2ce-89fe-42ff-96d4-00d1edbe2aef" (UID: "057fc2ce-89fe-42ff-96d4-00d1edbe2aef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 01:59:15.224414 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:15.224398 6221 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/057fc2ce-89fe-42ff-96d4-00d1edbe2aef-routing-overlay-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:59:15.224456 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:15.224417 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/057fc2ce-89fe-42ff-96d4-00d1edbe2aef-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:59:15.225823 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:15.225801 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/057fc2ce-89fe-42ff-96d4-00d1edbe2aef-provider-credentials" pod "057fc2ce-89fe-42ff-96d4-00d1edbe2aef" (UID: "057fc2ce-89fe-42ff-96d4-00d1edbe2aef"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:59:15.225912 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:15.225830 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/057fc2ce-89fe-42ff-96d4-00d1edbe2aef-kube-api-access-47mtz" pod "057fc2ce-89fe-42ff-96d4-00d1edbe2aef" (UID: "057fc2ce-89fe-42ff-96d4-00d1edbe2aef"). InnerVolumeSpecName "kube-api-access-47mtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 01:59:15.328816 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:15.327861 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47mtz\" (UniqueName: \"kubernetes.io/projected/057fc2ce-89fe-42ff-96d4-00d1edbe2aef-kube-api-access-47mtz\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:59:15.328816 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:15.327916 6221 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/057fc2ce-89fe-42ff-96d4-00d1edbe2aef-provider-credentials\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 01:59:15.948570 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:15.948539 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-78572h7dk" Oct 30 01:59:15.949592 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:15.948654 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-78572h7dk" event={"ID":"057fc2ce-89fe-42ff-96d4-00d1edbe2aef","Type":"ContainerDied","Data":"61a0ec5fa3a89aaee5601a45ed3db765fe7f126ef98f2fa3b0c2687381258020"} Oct 30 01:59:15.949719 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:15.949634 6221 scope.go:122] "RemoveContainer" containerID="34b179226a9b427ad2005fae96d0a977f93d0602c6e8d8e505e9818893f3fe0b" Oct 30 01:59:16.005407 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:16.005371 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-78572h7dk"] Oct 30 01:59:16.015284 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:16.015251 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-78572h7dk"] Oct 30 01:59:16.553630 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:16.553595 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="057fc2ce-89fe-42ff-96d4-00d1edbe2aef" path="/var/lib/kubelet/pods/057fc2ce-89fe-42ff-96d4-00d1edbe2aef/volumes" Oct 30 01:59:21.495210 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:59:21.495161 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:59:34.669259 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.669225 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8mnlq"] Oct 30 01:59:34.669960 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.669922 6221 cpu_manager.go:414] "containerMap: removing container" podUID="cb0d5c5a-987f-4b86-b072-3147e343d83d" containerName="payload-processing" Oct 30 01:59:34.669960 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.669956 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb0d5c5a-987f-4b86-b072-3147e343d83d" containerName="payload-processing" Oct 30 01:59:34.670125 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.669982 6221 cpu_manager.go:414] "containerMap: removing container" podUID="057fc2ce-89fe-42ff-96d4-00d1edbe2aef" containerName="payload-processing" Oct 30 01:59:34.670125 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.669988 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="057fc2ce-89fe-42ff-96d4-00d1edbe2aef" containerName="payload-processing" Oct 30 01:59:34.670125 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.670114 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="057fc2ce-89fe-42ff-96d4-00d1edbe2aef" containerName="payload-processing" Oct 30 01:59:34.670222 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.670129 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb0d5c5a-987f-4b86-b072-3147e343d83d" containerName="payload-processing" Oct 30 01:59:34.674107 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.674086 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8mnlq" Oct 30 01:59:34.689714 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.689689 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-s9nfd\"" Oct 30 01:59:34.691324 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.691300 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs"] Oct 30 01:59:34.695838 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.695815 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs" Oct 30 01:59:34.704374 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.704344 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8mnlq"] Oct 30 01:59:34.718972 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.718936 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs"] Oct 30 01:59:34.837715 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.837682 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/161c7dc2-54ae-4cb9-9105-3035c5f4875f-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bqtxs\" (UID: \"161c7dc2-54ae-4cb9-9105-3035c5f4875f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs" Oct 30 01:59:34.837930 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.837736 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8mnlq\" (UID: \"0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8mnlq" Oct 30 01:59:34.837930 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.837759 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/161c7dc2-54ae-4cb9-9105-3035c5f4875f-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bqtxs\" (UID: \"161c7dc2-54ae-4cb9-9105-3035c5f4875f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs" Oct 30 01:59:34.837930 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.837812 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/161c7dc2-54ae-4cb9-9105-3035c5f4875f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bqtxs\" (UID: \"161c7dc2-54ae-4cb9-9105-3035c5f4875f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs" Oct 30 01:59:34.837930 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.837911 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8mnlq\" (UID: \"0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8mnlq" Oct 30 01:59:34.838187 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.837950 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8mnlq\" (UID: \"0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8mnlq" Oct 30 01:59:34.838187 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.837975 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cdml\" (UniqueName: \"kubernetes.io/projected/0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4-kube-api-access-6cdml\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8mnlq\" (UID: \"0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8mnlq" Oct 30 01:59:34.838187 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.838145 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jptk\" (UniqueName: \"kubernetes.io/projected/161c7dc2-54ae-4cb9-9105-3035c5f4875f-kube-api-access-5jptk\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bqtxs\" (UID: \"161c7dc2-54ae-4cb9-9105-3035c5f4875f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs" Oct 30 01:59:34.838366 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.838226 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8mnlq\" (UID: \"0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8mnlq" Oct 30 01:59:34.838366 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.838305 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/161c7dc2-54ae-4cb9-9105-3035c5f4875f-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bqtxs\" (UID: \"161c7dc2-54ae-4cb9-9105-3035c5f4875f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs" Oct 30 01:59:34.838366 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.838348 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/161c7dc2-54ae-4cb9-9105-3035c5f4875f-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bqtxs\" (UID: \"161c7dc2-54ae-4cb9-9105-3035c5f4875f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs" Oct 30 01:59:34.838471 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.838396 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8mnlq\" (UID: \"0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8mnlq" Oct 30 01:59:34.838471 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.838447 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8mnlq\" (UID: \"0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8mnlq" Oct 30 01:59:34.838471 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.838466 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/161c7dc2-54ae-4cb9-9105-3035c5f4875f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bqtxs\" (UID: \"161c7dc2-54ae-4cb9-9105-3035c5f4875f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs" Oct 30 01:59:34.838647 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.838591 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/161c7dc2-54ae-4cb9-9105-3035c5f4875f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bqtxs\" (UID: \"161c7dc2-54ae-4cb9-9105-3035c5f4875f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs" Oct 30 01:59:34.838647 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.838624 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/161c7dc2-54ae-4cb9-9105-3035c5f4875f-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bqtxs\" (UID: \"161c7dc2-54ae-4cb9-9105-3035c5f4875f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs" Oct 30 01:59:34.838769 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.838665 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8mnlq\" (UID: \"0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8mnlq" Oct 30 01:59:34.838769 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.838747 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8mnlq\" (UID: \"0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8mnlq" Oct 30 01:59:34.940129 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.940028 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8mnlq\" (UID: \"0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8mnlq" Oct 30 01:59:34.940129 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.940073 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8mnlq\" (UID: \"0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8mnlq" Oct 30 01:59:34.940129 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.940106 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/161c7dc2-54ae-4cb9-9105-3035c5f4875f-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bqtxs\" (UID: \"161c7dc2-54ae-4cb9-9105-3035c5f4875f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs" Oct 30 01:59:34.940442 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.940155 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8mnlq\" (UID: \"0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8mnlq" Oct 30 01:59:34.940442 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.940186 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/161c7dc2-54ae-4cb9-9105-3035c5f4875f-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bqtxs\" (UID: \"161c7dc2-54ae-4cb9-9105-3035c5f4875f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs" Oct 30 01:59:34.940442 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.940210 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/161c7dc2-54ae-4cb9-9105-3035c5f4875f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bqtxs\" (UID: \"161c7dc2-54ae-4cb9-9105-3035c5f4875f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs" Oct 30 01:59:34.940442 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.940249 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8mnlq\" (UID: \"0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8mnlq" Oct 30 01:59:34.940442 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.940280 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8mnlq\" (UID: \"0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8mnlq" Oct 30 01:59:34.940442 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.940303 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cdml\" (UniqueName: \"kubernetes.io/projected/0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4-kube-api-access-6cdml\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8mnlq\" (UID: \"0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8mnlq" Oct 30 01:59:34.940833 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.940499 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jptk\" (UniqueName: \"kubernetes.io/projected/161c7dc2-54ae-4cb9-9105-3035c5f4875f-kube-api-access-5jptk\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bqtxs\" (UID: \"161c7dc2-54ae-4cb9-9105-3035c5f4875f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs" Oct 30 01:59:34.940833 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.940588 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8mnlq\" (UID: \"0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8mnlq" Oct 30 01:59:34.940833 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.940604 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8mnlq\" (UID: \"0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8mnlq" Oct 30 01:59:34.940833 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.940627 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/161c7dc2-54ae-4cb9-9105-3035c5f4875f-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bqtxs\" (UID: \"161c7dc2-54ae-4cb9-9105-3035c5f4875f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs" Oct 30 01:59:34.940833 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.940653 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/161c7dc2-54ae-4cb9-9105-3035c5f4875f-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bqtxs\" (UID: \"161c7dc2-54ae-4cb9-9105-3035c5f4875f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs" Oct 30 01:59:34.940833 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.940707 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/161c7dc2-54ae-4cb9-9105-3035c5f4875f-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bqtxs\" (UID: \"161c7dc2-54ae-4cb9-9105-3035c5f4875f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs" Oct 30 01:59:34.940833 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.940735 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8mnlq\" (UID: \"0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8mnlq" Oct 30 01:59:34.940833 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.940776 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8mnlq\" (UID: \"0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8mnlq" Oct 30 01:59:34.940833 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.940802 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/161c7dc2-54ae-4cb9-9105-3035c5f4875f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bqtxs\" (UID: \"161c7dc2-54ae-4cb9-9105-3035c5f4875f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs" Oct 30 01:59:34.941397 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.940880 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/161c7dc2-54ae-4cb9-9105-3035c5f4875f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bqtxs\" (UID: \"161c7dc2-54ae-4cb9-9105-3035c5f4875f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs" Oct 30 01:59:34.941397 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.940901 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8mnlq\" (UID: \"0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8mnlq" Oct 30 01:59:34.941397 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.940916 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/161c7dc2-54ae-4cb9-9105-3035c5f4875f-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bqtxs\" (UID: \"161c7dc2-54ae-4cb9-9105-3035c5f4875f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs" Oct 30 01:59:34.941397 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.940969 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8mnlq\" (UID: \"0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8mnlq" Oct 30 01:59:34.941397 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.941079 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/161c7dc2-54ae-4cb9-9105-3035c5f4875f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bqtxs\" (UID: \"161c7dc2-54ae-4cb9-9105-3035c5f4875f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs" Oct 30 01:59:34.941397 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.941199 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/161c7dc2-54ae-4cb9-9105-3035c5f4875f-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bqtxs\" (UID: \"161c7dc2-54ae-4cb9-9105-3035c5f4875f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs" Oct 30 01:59:34.941397 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.941281 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8mnlq\" (UID: \"0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8mnlq" Oct 30 01:59:34.941397 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.941371 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/161c7dc2-54ae-4cb9-9105-3035c5f4875f-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bqtxs\" (UID: \"161c7dc2-54ae-4cb9-9105-3035c5f4875f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs" Oct 30 01:59:34.941768 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.941458 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/161c7dc2-54ae-4cb9-9105-3035c5f4875f-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bqtxs\" (UID: \"161c7dc2-54ae-4cb9-9105-3035c5f4875f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs" Oct 30 01:59:34.941768 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.941519 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8mnlq\" (UID: \"0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8mnlq" Oct 30 01:59:34.942882 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.942862 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8mnlq\" (UID: \"0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8mnlq" Oct 30 01:59:34.943364 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.943333 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8mnlq\" (UID: \"0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8mnlq" Oct 30 01:59:34.943559 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.943512 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/161c7dc2-54ae-4cb9-9105-3035c5f4875f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bqtxs\" (UID: \"161c7dc2-54ae-4cb9-9105-3035c5f4875f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs" Oct 30 01:59:34.943726 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.943711 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/161c7dc2-54ae-4cb9-9105-3035c5f4875f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bqtxs\" (UID: \"161c7dc2-54ae-4cb9-9105-3035c5f4875f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs" Oct 30 01:59:34.961625 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.961593 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8mnlq\" (UID: \"0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8mnlq" Oct 30 01:59:34.961724 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.961683 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cdml\" (UniqueName: \"kubernetes.io/projected/0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4-kube-api-access-6cdml\") pod \"maas-default-gateway-openshift-default-5bc884dd87-8mnlq\" (UID: \"0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8mnlq" Oct 30 01:59:34.961807 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.961786 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/161c7dc2-54ae-4cb9-9105-3035c5f4875f-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bqtxs\" (UID: \"161c7dc2-54ae-4cb9-9105-3035c5f4875f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs" Oct 30 01:59:34.963012 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.962996 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jptk\" (UniqueName: \"kubernetes.io/projected/161c7dc2-54ae-4cb9-9105-3035c5f4875f-kube-api-access-5jptk\") pod \"maas-default-gateway-openshift-default-5bc884dd87-bqtxs\" (UID: \"161c7dc2-54ae-4cb9-9105-3035c5f4875f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs" Oct 30 01:59:34.988819 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:34.988796 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8mnlq" Oct 30 01:59:35.027318 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:35.027281 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs" Oct 30 01:59:35.314139 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:35.314106 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8mnlq"] Oct 30 01:59:35.315672 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:59:35.315640 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f60b8d7_d3d0_4ce0_bd5b_28bd7afaadf4.slice/crio-21618e3ab178a6aa2cbb13903293c16f13d97ababd1fcbff10501a03896bd88d WatchSource:0}: Error finding container 21618e3ab178a6aa2cbb13903293c16f13d97ababd1fcbff10501a03896bd88d: Status 404 returned error can't find the container with id 21618e3ab178a6aa2cbb13903293c16f13d97ababd1fcbff10501a03896bd88d Oct 30 01:59:35.318494 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:35.318461 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:59:35.318602 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:35.318525 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:59:35.318602 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:35.318587 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:59:35.375936 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:35.375903 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs"] Oct 30 01:59:35.408721 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:35.405563 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:59:35.408721 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:35.405659 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:59:35.408721 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:35.405699 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:59:36.100870 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:36.100833 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8mnlq" event={"ID":"0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4","Type":"ContainerStarted","Data":"a907f67df685742bb550be94ae2cc1680bf18cacd91790fc1c24bd1bc0f2ad11"} Oct 30 01:59:36.101372 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:36.100890 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8mnlq" event={"ID":"0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4","Type":"ContainerStarted","Data":"21618e3ab178a6aa2cbb13903293c16f13d97ababd1fcbff10501a03896bd88d"} Oct 30 01:59:36.102592 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:36.102563 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs" event={"ID":"161c7dc2-54ae-4cb9-9105-3035c5f4875f","Type":"ContainerStarted","Data":"8fc682e71c34336824667b2961312a35a807973ca69017380a267ccf2f509abe"} Oct 30 01:59:36.102705 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:36.102612 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs" event={"ID":"161c7dc2-54ae-4cb9-9105-3035c5f4875f","Type":"ContainerStarted","Data":"bfd00aaecb02b812222fdb3f094a814b7efc53908600280de596a3dc6709be91"} Oct 30 01:59:36.171246 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:36.171171 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8mnlq" podStartSLOduration=2.171155971 podStartE2EDuration="2.171155971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:59:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:59:36.16975551 +0000 UTC m=+2499.748042843" watchObservedRunningTime="2026-10-30 01:59:36.171155971 +0000 UTC m=+2499.749443307" Oct 30 01:59:36.259451 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:36.259391 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs" podStartSLOduration=2.259374891 podStartE2EDuration="2.259374891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:59:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:59:36.216047327 +0000 UTC m=+2499.794334665" watchObservedRunningTime="2026-10-30 01:59:36.259374891 +0000 UTC m=+2499.837662286" Oct 30 01:59:36.989157 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:36.989122 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8mnlq" Oct 30 01:59:36.990568 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:36.990519 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-8mnlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 01:59:36.990697 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:36.990589 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8mnlq" podUID="0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 01:59:37.031557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:37.031141 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs" Oct 30 01:59:37.031557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:37.031316 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bqtxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 30 01:59:37.031557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:37.031363 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs" podUID="161c7dc2-54ae-4cb9-9105-3035c5f4875f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 30 01:59:37.989180 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:37.989147 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-8mnlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 01:59:37.989678 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:37.989206 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8mnlq" podUID="0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 01:59:38.040557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:38.040216 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bqtxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 30 01:59:38.040557 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:38.040285 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs" podUID="161c7dc2-54ae-4cb9-9105-3035c5f4875f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 30 01:59:38.990108 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:38.990070 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-8mnlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 01:59:38.990522 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:38.990131 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8mnlq" podUID="0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 01:59:39.028558 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:39.028131 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bqtxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 30 01:59:39.028558 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:39.028205 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs" podUID="161c7dc2-54ae-4cb9-9105-3035c5f4875f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 30 01:59:39.989597 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:39.989563 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-8mnlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 01:59:39.989776 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:39.989620 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8mnlq" podUID="0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 01:59:40.030950 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:40.030910 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bqtxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 30 01:59:40.031580 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:40.031525 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs" podUID="161c7dc2-54ae-4cb9-9105-3035c5f4875f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 30 01:59:40.989381 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:40.989339 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-8mnlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 01:59:40.989614 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:40.989404 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8mnlq" podUID="0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 01:59:41.029010 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:41.028970 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bqtxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 30 01:59:41.029619 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:41.029580 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs" podUID="161c7dc2-54ae-4cb9-9105-3035c5f4875f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 30 01:59:41.989765 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:41.989721 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-8mnlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 01:59:41.990250 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:41.989790 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8mnlq" podUID="0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 01:59:42.028582 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:42.028511 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bqtxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 30 01:59:42.028762 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:42.028610 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs" podUID="161c7dc2-54ae-4cb9-9105-3035c5f4875f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 30 01:59:43.004735 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:43.004686 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-8mnlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" start-of-body= Oct 30 01:59:43.005313 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:43.004783 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8mnlq" podUID="0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.186:15021/healthz/ready\": dial tcp 10.134.0.186:15021: connect: connection refused" Oct 30 01:59:43.027925 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:43.027882 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-bqtxs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" start-of-body= Oct 30 01:59:43.028119 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:43.027940 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs" podUID="161c7dc2-54ae-4cb9-9105-3035c5f4875f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.187:15021/healthz/ready\": dial tcp 10.134.0.187:15021: connect: connection refused" Oct 30 01:59:44.158936 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:44.158894 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8mnlq" Oct 30 01:59:44.160287 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:44.160267 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8mnlq" Oct 30 01:59:44.162185 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:44.162161 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-8mnlq" Oct 30 01:59:44.234778 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:44.234744 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs" Oct 30 01:59:44.235714 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:44.235521 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs" Oct 30 01:59:44.236932 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:44.236913 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs" Oct 30 01:59:49.732279 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:49.732241 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k8czs"] Oct 30 01:59:49.737785 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:49.737761 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k8czs" Oct 30 01:59:49.760393 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:49.760362 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k8czs"] Oct 30 01:59:49.813595 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:49.813563 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7614c59-c623-460a-adae-dfe0601605dd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k8czs\" (UID: \"b7614c59-c623-460a-adae-dfe0601605dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k8czs" Oct 30 01:59:49.813777 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:49.813622 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7614c59-c623-460a-adae-dfe0601605dd-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k8czs\" (UID: \"b7614c59-c623-460a-adae-dfe0601605dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k8czs" Oct 30 01:59:49.813832 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:49.813792 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7614c59-c623-460a-adae-dfe0601605dd-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k8czs\" (UID: \"b7614c59-c623-460a-adae-dfe0601605dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k8czs" Oct 30 01:59:49.813936 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:49.813917 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7614c59-c623-460a-adae-dfe0601605dd-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k8czs\" (UID: \"b7614c59-c623-460a-adae-dfe0601605dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k8czs" Oct 30 01:59:49.813983 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:49.813968 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kqzt\" (UniqueName: \"kubernetes.io/projected/b7614c59-c623-460a-adae-dfe0601605dd-kube-api-access-4kqzt\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k8czs\" (UID: \"b7614c59-c623-460a-adae-dfe0601605dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k8czs" Oct 30 01:59:49.814021 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:49.814003 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7614c59-c623-460a-adae-dfe0601605dd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k8czs\" (UID: \"b7614c59-c623-460a-adae-dfe0601605dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k8czs" Oct 30 01:59:49.814106 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:49.814091 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7614c59-c623-460a-adae-dfe0601605dd-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k8czs\" (UID: \"b7614c59-c623-460a-adae-dfe0601605dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k8czs" Oct 30 01:59:49.814234 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:49.814220 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7614c59-c623-460a-adae-dfe0601605dd-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k8czs\" (UID: \"b7614c59-c623-460a-adae-dfe0601605dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k8czs" Oct 30 01:59:49.814281 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:49.814267 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7614c59-c623-460a-adae-dfe0601605dd-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k8czs\" (UID: \"b7614c59-c623-460a-adae-dfe0601605dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k8czs" Oct 30 01:59:49.915313 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:49.915280 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7614c59-c623-460a-adae-dfe0601605dd-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k8czs\" (UID: \"b7614c59-c623-460a-adae-dfe0601605dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k8czs" Oct 30 01:59:49.915502 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:49.915388 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7614c59-c623-460a-adae-dfe0601605dd-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k8czs\" (UID: \"b7614c59-c623-460a-adae-dfe0601605dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k8czs" Oct 30 01:59:49.915502 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:49.915430 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7614c59-c623-460a-adae-dfe0601605dd-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k8czs\" (UID: \"b7614c59-c623-460a-adae-dfe0601605dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k8czs" Oct 30 01:59:49.915502 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:49.915458 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7614c59-c623-460a-adae-dfe0601605dd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k8czs\" (UID: \"b7614c59-c623-460a-adae-dfe0601605dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k8czs" Oct 30 01:59:49.915698 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:49.915516 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7614c59-c623-460a-adae-dfe0601605dd-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k8czs\" (UID: \"b7614c59-c623-460a-adae-dfe0601605dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k8czs" Oct 30 01:59:49.915698 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:49.915672 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7614c59-c623-460a-adae-dfe0601605dd-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k8czs\" (UID: \"b7614c59-c623-460a-adae-dfe0601605dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k8czs" Oct 30 01:59:49.915820 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:49.915751 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7614c59-c623-460a-adae-dfe0601605dd-credential-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k8czs\" (UID: \"b7614c59-c623-460a-adae-dfe0601605dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k8czs" Oct 30 01:59:49.915820 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:49.915781 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7614c59-c623-460a-adae-dfe0601605dd-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k8czs\" (UID: \"b7614c59-c623-460a-adae-dfe0601605dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k8czs" Oct 30 01:59:49.915820 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:49.915799 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7614c59-c623-460a-adae-dfe0601605dd-workload-certs\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k8czs\" (UID: \"b7614c59-c623-460a-adae-dfe0601605dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k8czs" Oct 30 01:59:49.915967 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:49.915821 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kqzt\" (UniqueName: \"kubernetes.io/projected/b7614c59-c623-460a-adae-dfe0601605dd-kube-api-access-4kqzt\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k8czs\" (UID: \"b7614c59-c623-460a-adae-dfe0601605dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k8czs" Oct 30 01:59:49.915967 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:49.915851 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7614c59-c623-460a-adae-dfe0601605dd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k8czs\" (UID: \"b7614c59-c623-460a-adae-dfe0601605dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k8czs" Oct 30 01:59:49.915967 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:49.915870 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7614c59-c623-460a-adae-dfe0601605dd-workload-socket\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k8czs\" (UID: \"b7614c59-c623-460a-adae-dfe0601605dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k8czs" Oct 30 01:59:49.916074 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:49.916034 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7614c59-c623-460a-adae-dfe0601605dd-istio-data\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k8czs\" (UID: \"b7614c59-c623-460a-adae-dfe0601605dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k8czs" Oct 30 01:59:49.916287 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:49.916265 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7614c59-c623-460a-adae-dfe0601605dd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k8czs\" (UID: \"b7614c59-c623-460a-adae-dfe0601605dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k8czs" Oct 30 01:59:49.917967 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:49.917945 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7614c59-c623-460a-adae-dfe0601605dd-istio-envoy\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k8czs\" (UID: \"b7614c59-c623-460a-adae-dfe0601605dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k8czs" Oct 30 01:59:49.918235 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:49.918216 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7614c59-c623-460a-adae-dfe0601605dd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k8czs\" (UID: \"b7614c59-c623-460a-adae-dfe0601605dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k8czs" Oct 30 01:59:49.944172 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:49.944137 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7614c59-c623-460a-adae-dfe0601605dd-istio-token\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k8czs\" (UID: \"b7614c59-c623-460a-adae-dfe0601605dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k8czs" Oct 30 01:59:49.945273 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:49.945251 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kqzt\" (UniqueName: \"kubernetes.io/projected/b7614c59-c623-460a-adae-dfe0601605dd-kube-api-access-4kqzt\") pod \"maas-default-gateway-openshift-default-5bc884dd87-k8czs\" (UID: \"b7614c59-c623-460a-adae-dfe0601605dd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k8czs" Oct 30 01:59:50.062443 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:50.062407 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k8czs" Oct 30 01:59:50.388118 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:50.388091 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k8czs"] Oct 30 01:59:50.390671 ip-10-0-135-65 kubenswrapper[6221]: W1030 01:59:50.390645 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7614c59_c623_460a_adae_dfe0601605dd.slice/crio-2aa8d56e16df80fa9fd45f61bebf3a0767e02841df51b2ee178636908fe3dcb0 WatchSource:0}: Error finding container 2aa8d56e16df80fa9fd45f61bebf3a0767e02841df51b2ee178636908fe3dcb0: Status 404 returned error can't find the container with id 2aa8d56e16df80fa9fd45f61bebf3a0767e02841df51b2ee178636908fe3dcb0 Oct 30 01:59:50.393678 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:50.393649 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:59:50.393748 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:50.393712 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:59:50.393786 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:50.393758 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 01:59:51.305171 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:51.305127 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k8czs" event={"ID":"b7614c59-c623-460a-adae-dfe0601605dd","Type":"ContainerStarted","Data":"286ee107ee6e7e0d457081b48f6c265a5afb8d309ecf76777ef802b85aa85ef9"} Oct 30 01:59:51.305665 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:51.305216 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k8czs" event={"ID":"b7614c59-c623-460a-adae-dfe0601605dd","Type":"ContainerStarted","Data":"2aa8d56e16df80fa9fd45f61bebf3a0767e02841df51b2ee178636908fe3dcb0"} Oct 30 01:59:51.365093 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:51.364990 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k8czs" podStartSLOduration=2.364973336 podStartE2EDuration="2.364973336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 01:59:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 01:59:51.362070811 +0000 UTC m=+2514.940358149" watchObservedRunningTime="2026-10-30 01:59:51.364973336 +0000 UTC m=+2514.943260672" Oct 30 01:59:51.496300 ip-10-0-135-65 kubenswrapper[6221]: E1030 01:59:51.496264 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 01:59:52.062883 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:52.062846 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k8czs" Oct 30 01:59:52.064837 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:52.064800 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-k8czs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 30 01:59:52.064984 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:52.064862 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k8czs" podUID="b7614c59-c623-460a-adae-dfe0601605dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 30 01:59:52.884259 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:52.884222 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 01:59:53.062842 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:53.062804 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-k8czs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 30 01:59:53.063012 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:53.062862 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k8czs" podUID="b7614c59-c623-460a-adae-dfe0601605dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 30 01:59:54.063422 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:54.063387 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-k8czs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 30 01:59:54.064004 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:54.063470 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k8czs" podUID="b7614c59-c623-460a-adae-dfe0601605dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 30 01:59:55.063299 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:55.063262 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-k8czs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 30 01:59:55.063484 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:55.063315 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k8czs" podUID="b7614c59-c623-460a-adae-dfe0601605dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 30 01:59:56.063051 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:56.063012 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-k8czs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 30 01:59:56.063279 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:56.063074 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k8czs" podUID="b7614c59-c623-460a-adae-dfe0601605dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 30 01:59:57.062841 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:57.062801 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-k8czs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 30 01:59:57.063233 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:57.062871 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k8czs" podUID="b7614c59-c623-460a-adae-dfe0601605dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 30 01:59:58.062940 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:58.062908 6221 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5bc884dd87-k8czs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body= Oct 30 01:59:58.063337 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:58.062963 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k8czs" podUID="b7614c59-c623-460a-adae-dfe0601605dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" Oct 30 01:59:59.453848 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:59.453813 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k8czs" Oct 30 01:59:59.455078 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:59.455054 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k8czs" Oct 30 01:59:59.457703 ip-10-0-135-65 kubenswrapper[6221]: I1030 01:59:59.457681 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-k8czs" Oct 30 02:00:00.224310 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.224263 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888760-jzwd6"] Oct 30 02:00:00.232369 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.232343 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888760-nbm5m"] Oct 30 02:00:00.233823 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.233682 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888760-jzwd6" Oct 30 02:00:00.241766 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.241744 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888760-nbm5m" Oct 30 02:00:00.246327 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.246300 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888760-qmg68"] Oct 30 02:00:00.251587 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.251523 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888760-qmg68" Oct 30 02:00:00.253190 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.253164 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888760-jzwd6"] Oct 30 02:00:00.258627 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.258598 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888760-nbm5m"] Oct 30 02:00:00.263183 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.259598 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:00:00.271583 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.269958 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888760-lfdmp"] Oct 30 02:00:00.275995 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.275977 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888760-lfdmp" Oct 30 02:00:00.305066 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.305027 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f6e11e-29888760-sx6jd"] Oct 30 02:00:00.314483 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.314442 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f6e11e-29888760-sx6jd" Oct 30 02:00:00.318742 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.318063 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888760-lfdmp"] Oct 30 02:00:00.324852 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.324784 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888760-qmg68"] Oct 30 02:00:00.341120 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.341081 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1038f98-dfbd-4839-b07f-2cb9742f144c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888760-nbm5m\" (UID: \"f1038f98-dfbd-4839-b07f-2cb9742f144c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888760-nbm5m" Oct 30 02:00:00.341277 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.341137 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb7ts\" (UniqueName: \"kubernetes.io/projected/a451aa96-5403-409f-8a18-c1b2ea5a2231-kube-api-access-mb7ts\") pod \"maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888760-lfdmp\" (UID: \"a451aa96-5403-409f-8a18-c1b2ea5a2231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888760-lfdmp" Oct 30 02:00:00.341371 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.341290 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlfnp\" (UniqueName: \"kubernetes.io/projected/381d1132-541d-4397-91ab-285a4b873991-kube-api-access-nlfnp\") pod \"maas-api-key-cleanup-e2e-extmod-b-2e618f-29888760-qmg68\" (UID: \"381d1132-541d-4397-91ab-285a4b873991\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888760-qmg68" Oct 30 02:00:00.341371 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.341335 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgg45\" (UniqueName: \"kubernetes.io/projected/f1038f98-dfbd-4839-b07f-2cb9742f144c-kube-api-access-xgg45\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888760-nbm5m\" (UID: \"f1038f98-dfbd-4839-b07f-2cb9742f144c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888760-nbm5m" Oct 30 02:00:00.341543 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.341514 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a451aa96-5403-409f-8a18-c1b2ea5a2231-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888760-lfdmp\" (UID: \"a451aa96-5403-409f-8a18-c1b2ea5a2231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888760-lfdmp" Oct 30 02:00:00.341610 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.341591 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xqrj\" (UniqueName: \"kubernetes.io/projected/76d7ecb2-375a-49ae-a766-db2b5405bfcd-kube-api-access-9xqrj\") pod \"maas-api-key-cleanup-e2e-extmod-b-cd8173-29888760-jzwd6\" (UID: \"76d7ecb2-375a-49ae-a766-db2b5405bfcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888760-jzwd6" Oct 30 02:00:00.341689 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.341671 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/381d1132-541d-4397-91ab-285a4b873991-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2e618f-29888760-qmg68\" (UID: \"381d1132-541d-4397-91ab-285a4b873991\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888760-qmg68" Oct 30 02:00:00.341752 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.341733 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76d7ecb2-375a-49ae-a766-db2b5405bfcd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-cd8173-29888760-jzwd6\" (UID: \"76d7ecb2-375a-49ae-a766-db2b5405bfcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888760-jzwd6" Oct 30 02:00:00.352326 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.352301 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f6e11e-29888760-sx6jd"] Oct 30 02:00:00.443654 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.443619 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlfnp\" (UniqueName: \"kubernetes.io/projected/381d1132-541d-4397-91ab-285a4b873991-kube-api-access-nlfnp\") pod \"maas-api-key-cleanup-e2e-extmod-b-2e618f-29888760-qmg68\" (UID: \"381d1132-541d-4397-91ab-285a4b873991\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888760-qmg68" Oct 30 02:00:00.443850 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.443667 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgg45\" (UniqueName: \"kubernetes.io/projected/f1038f98-dfbd-4839-b07f-2cb9742f144c-kube-api-access-xgg45\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888760-nbm5m\" (UID: \"f1038f98-dfbd-4839-b07f-2cb9742f144c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888760-nbm5m" Oct 30 02:00:00.443850 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.443811 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a451aa96-5403-409f-8a18-c1b2ea5a2231-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888760-lfdmp\" (UID: \"a451aa96-5403-409f-8a18-c1b2ea5a2231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888760-lfdmp" Oct 30 02:00:00.443939 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.443858 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xqrj\" (UniqueName: \"kubernetes.io/projected/76d7ecb2-375a-49ae-a766-db2b5405bfcd-kube-api-access-9xqrj\") pod \"maas-api-key-cleanup-e2e-extmod-b-cd8173-29888760-jzwd6\" (UID: \"76d7ecb2-375a-49ae-a766-db2b5405bfcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888760-jzwd6" Oct 30 02:00:00.443939 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.443922 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/381d1132-541d-4397-91ab-285a4b873991-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2e618f-29888760-qmg68\" (UID: \"381d1132-541d-4397-91ab-285a4b873991\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888760-qmg68" Oct 30 02:00:00.444027 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.443976 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76d7ecb2-375a-49ae-a766-db2b5405bfcd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-cd8173-29888760-jzwd6\" (UID: \"76d7ecb2-375a-49ae-a766-db2b5405bfcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888760-jzwd6" Oct 30 02:00:00.444079 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.444062 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn5d4\" (UniqueName: \"kubernetes.io/projected/108b6f30-ce76-41e2-a888-1d5601c4eeea-kube-api-access-rn5d4\") pod \"maas-api-key-cleanup-e2e-extmod-b-f6e11e-29888760-sx6jd\" (UID: \"108b6f30-ce76-41e2-a888-1d5601c4eeea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f6e11e-29888760-sx6jd" Oct 30 02:00:00.444134 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.444109 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1038f98-dfbd-4839-b07f-2cb9742f144c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888760-nbm5m\" (UID: \"f1038f98-dfbd-4839-b07f-2cb9742f144c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888760-nbm5m" Oct 30 02:00:00.444214 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.444150 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/108b6f30-ce76-41e2-a888-1d5601c4eeea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-f6e11e-29888760-sx6jd\" (UID: \"108b6f30-ce76-41e2-a888-1d5601c4eeea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f6e11e-29888760-sx6jd" Oct 30 02:00:00.444214 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.444181 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb7ts\" (UniqueName: \"kubernetes.io/projected/a451aa96-5403-409f-8a18-c1b2ea5a2231-kube-api-access-mb7ts\") pod \"maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888760-lfdmp\" (UID: \"a451aa96-5403-409f-8a18-c1b2ea5a2231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888760-lfdmp" Oct 30 02:00:00.444723 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.444698 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a451aa96-5403-409f-8a18-c1b2ea5a2231-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888760-lfdmp\" (UID: \"a451aa96-5403-409f-8a18-c1b2ea5a2231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888760-lfdmp" Oct 30 02:00:00.444903 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.444879 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76d7ecb2-375a-49ae-a766-db2b5405bfcd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-cd8173-29888760-jzwd6\" (UID: \"76d7ecb2-375a-49ae-a766-db2b5405bfcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888760-jzwd6" Oct 30 02:00:00.445154 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.445133 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1038f98-dfbd-4839-b07f-2cb9742f144c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888760-nbm5m\" (UID: \"f1038f98-dfbd-4839-b07f-2cb9742f144c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888760-nbm5m" Oct 30 02:00:00.445366 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.445347 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/381d1132-541d-4397-91ab-285a4b873991-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2e618f-29888760-qmg68\" (UID: \"381d1132-541d-4397-91ab-285a4b873991\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888760-qmg68" Oct 30 02:00:00.482006 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.481901 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgg45\" (UniqueName: \"kubernetes.io/projected/f1038f98-dfbd-4839-b07f-2cb9742f144c-kube-api-access-xgg45\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888760-nbm5m\" (UID: \"f1038f98-dfbd-4839-b07f-2cb9742f144c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888760-nbm5m" Oct 30 02:00:00.487778 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.487749 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlfnp\" (UniqueName: \"kubernetes.io/projected/381d1132-541d-4397-91ab-285a4b873991-kube-api-access-nlfnp\") pod \"maas-api-key-cleanup-e2e-extmod-b-2e618f-29888760-qmg68\" (UID: \"381d1132-541d-4397-91ab-285a4b873991\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888760-qmg68" Oct 30 02:00:00.488291 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.488268 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xqrj\" (UniqueName: \"kubernetes.io/projected/76d7ecb2-375a-49ae-a766-db2b5405bfcd-kube-api-access-9xqrj\") pod \"maas-api-key-cleanup-e2e-extmod-b-cd8173-29888760-jzwd6\" (UID: \"76d7ecb2-375a-49ae-a766-db2b5405bfcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888760-jzwd6" Oct 30 02:00:00.488706 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.488688 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb7ts\" (UniqueName: \"kubernetes.io/projected/a451aa96-5403-409f-8a18-c1b2ea5a2231-kube-api-access-mb7ts\") pod \"maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888760-lfdmp\" (UID: \"a451aa96-5403-409f-8a18-c1b2ea5a2231\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888760-lfdmp" Oct 30 02:00:00.545140 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.545100 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn5d4\" (UniqueName: \"kubernetes.io/projected/108b6f30-ce76-41e2-a888-1d5601c4eeea-kube-api-access-rn5d4\") pod \"maas-api-key-cleanup-e2e-extmod-b-f6e11e-29888760-sx6jd\" (UID: \"108b6f30-ce76-41e2-a888-1d5601c4eeea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f6e11e-29888760-sx6jd" Oct 30 02:00:00.545345 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.545180 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/108b6f30-ce76-41e2-a888-1d5601c4eeea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-f6e11e-29888760-sx6jd\" (UID: \"108b6f30-ce76-41e2-a888-1d5601c4eeea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f6e11e-29888760-sx6jd" Oct 30 02:00:00.546064 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.546040 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/108b6f30-ce76-41e2-a888-1d5601c4eeea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-f6e11e-29888760-sx6jd\" (UID: \"108b6f30-ce76-41e2-a888-1d5601c4eeea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f6e11e-29888760-sx6jd" Oct 30 02:00:00.564564 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.564498 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888760-jzwd6" Oct 30 02:00:00.566948 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.566926 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn5d4\" (UniqueName: \"kubernetes.io/projected/108b6f30-ce76-41e2-a888-1d5601c4eeea-kube-api-access-rn5d4\") pod \"maas-api-key-cleanup-e2e-extmod-b-f6e11e-29888760-sx6jd\" (UID: \"108b6f30-ce76-41e2-a888-1d5601c4eeea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f6e11e-29888760-sx6jd" Oct 30 02:00:00.579329 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.579301 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888760-nbm5m" Oct 30 02:00:00.624005 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.622789 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888760-qmg68" Oct 30 02:00:00.630606 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.630512 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888760-lfdmp" Oct 30 02:00:00.638268 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.638070 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f6e11e-29888760-sx6jd" Oct 30 02:00:00.891571 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.891527 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888760-jzwd6"] Oct 30 02:00:00.895012 ip-10-0-135-65 kubenswrapper[6221]: W1030 02:00:00.894979 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76d7ecb2_375a_49ae_a766_db2b5405bfcd.slice/crio-dbede8a07d209de4dcc03842d216627d59ca903a4355aa92beb7b1522d77e0f7 WatchSource:0}: Error finding container dbede8a07d209de4dcc03842d216627d59ca903a4355aa92beb7b1522d77e0f7: Status 404 returned error can't find the container with id dbede8a07d209de4dcc03842d216627d59ca903a4355aa92beb7b1522d77e0f7 Oct 30 02:00:00.897498 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:00.897474 6221 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 02:00:01.059422 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:01.059394 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888760-qmg68"] Oct 30 02:00:01.061599 ip-10-0-135-65 kubenswrapper[6221]: W1030 02:00:01.061571 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod381d1132_541d_4397_91ab_285a4b873991.slice/crio-fe6e7780638c77753e930699be93240ca42af3af64db626a21fea443920484ed WatchSource:0}: Error finding container fe6e7780638c77753e930699be93240ca42af3af64db626a21fea443920484ed: Status 404 returned error can't find the container with id fe6e7780638c77753e930699be93240ca42af3af64db626a21fea443920484ed Oct 30 02:00:01.095488 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:01.095379 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f6e11e-29888760-sx6jd"] Oct 30 02:00:01.098902 ip-10-0-135-65 kubenswrapper[6221]: W1030 02:00:01.098875 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod108b6f30_ce76_41e2_a888_1d5601c4eeea.slice/crio-3c390d3185b8b47b0b3725d8a4dcddc508ba69125d4f690ef053bf3a10ef37af WatchSource:0}: Error finding container 3c390d3185b8b47b0b3725d8a4dcddc508ba69125d4f690ef053bf3a10ef37af: Status 404 returned error can't find the container with id 3c390d3185b8b47b0b3725d8a4dcddc508ba69125d4f690ef053bf3a10ef37af Oct 30 02:00:01.099882 ip-10-0-135-65 kubenswrapper[6221]: W1030 02:00:01.099859 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda451aa96_5403_409f_8a18_c1b2ea5a2231.slice/crio-3e35eafb228efde72afec357a7d639f918aa366a49562c8718e51bb231391d18 WatchSource:0}: Error finding container 3e35eafb228efde72afec357a7d639f918aa366a49562c8718e51bb231391d18: Status 404 returned error can't find the container with id 3e35eafb228efde72afec357a7d639f918aa366a49562c8718e51bb231391d18 Oct 30 02:00:01.100128 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:01.100106 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888760-lfdmp"] Oct 30 02:00:01.114889 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:01.114867 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888760-nbm5m"] Oct 30 02:00:01.116669 ip-10-0-135-65 kubenswrapper[6221]: W1030 02:00:01.116644 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1038f98_dfbd_4839_b07f_2cb9742f144c.slice/crio-08c2fde73b68bd4feb091d550518745f6378e7e07bff01a128d31c0a52549c0e WatchSource:0}: Error finding container 08c2fde73b68bd4feb091d550518745f6378e7e07bff01a128d31c0a52549c0e: Status 404 returned error can't find the container with id 08c2fde73b68bd4feb091d550518745f6378e7e07bff01a128d31c0a52549c0e Oct 30 02:00:01.451453 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:01.451351 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888760-qmg68" event={"ID":"381d1132-541d-4397-91ab-285a4b873991","Type":"ContainerStarted","Data":"fe6e7780638c77753e930699be93240ca42af3af64db626a21fea443920484ed"} Oct 30 02:00:01.452893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:01.452865 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f6e11e-29888760-sx6jd" event={"ID":"108b6f30-ce76-41e2-a888-1d5601c4eeea","Type":"ContainerStarted","Data":"3c390d3185b8b47b0b3725d8a4dcddc508ba69125d4f690ef053bf3a10ef37af"} Oct 30 02:00:01.454058 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:01.454038 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888760-jzwd6" event={"ID":"76d7ecb2-375a-49ae-a766-db2b5405bfcd","Type":"ContainerStarted","Data":"dbede8a07d209de4dcc03842d216627d59ca903a4355aa92beb7b1522d77e0f7"} Oct 30 02:00:01.455098 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:01.455080 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888760-lfdmp" event={"ID":"a451aa96-5403-409f-8a18-c1b2ea5a2231","Type":"ContainerStarted","Data":"3e35eafb228efde72afec357a7d639f918aa366a49562c8718e51bb231391d18"} Oct 30 02:00:01.456237 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:01.456218 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888760-nbm5m" event={"ID":"f1038f98-dfbd-4839-b07f-2cb9742f144c","Type":"ContainerStarted","Data":"08c2fde73b68bd4feb091d550518745f6378e7e07bff01a128d31c0a52549c0e"} Oct 30 02:00:06.520466 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:06.520362 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888760-qmg68" event={"ID":"381d1132-541d-4397-91ab-285a4b873991","Type":"ContainerStarted","Data":"826a8c5954d0b5bdbb08cb55d0ddfd59c441cb2ad06b96e5313aaf8c5b777968"} Oct 30 02:00:06.522091 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:06.522060 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f6e11e-29888760-sx6jd" event={"ID":"108b6f30-ce76-41e2-a888-1d5601c4eeea","Type":"ContainerStarted","Data":"68bb88b705af3fb50adf62ae1194062ce7d0bc096b395a240ef4341c22074ad1"} Oct 30 02:00:06.523612 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:06.523588 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888760-jzwd6" event={"ID":"76d7ecb2-375a-49ae-a766-db2b5405bfcd","Type":"ContainerStarted","Data":"26df5370feb3f95873d1657683d36f37321764be97eb77f3139a60579a086193"} Oct 30 02:00:06.525131 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:06.525103 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888760-lfdmp" event={"ID":"a451aa96-5403-409f-8a18-c1b2ea5a2231","Type":"ContainerStarted","Data":"7dbb8c59dfc9e7f92c8234abb8f1852a6efb60ad19857849f6ac4f012f5be402"} Oct 30 02:00:06.526776 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:06.526750 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888760-nbm5m" event={"ID":"f1038f98-dfbd-4839-b07f-2cb9742f144c","Type":"ContainerStarted","Data":"87eb8d24cecc631c38cf67313c396739e84a4147abb5729ebdfd151b5054e7b6"} Oct 30 02:00:06.560973 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:06.560917 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888760-qmg68" podStartSLOduration=1.756840681 podStartE2EDuration="6.560903337s" totalImagesPullingTime="4.804062656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:06.558630148 +0000 UTC m=+2530.136917483" watchObservedRunningTime="2026-10-30 02:00:06.560903337 +0000 UTC m=+2530.139190718" Oct 30 02:00:06.603788 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:06.603734 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888760-jzwd6" podStartSLOduration=1.7887663919999999 podStartE2EDuration="6.603718996s" totalImagesPullingTime="4.814952604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:06.60043675 +0000 UTC m=+2530.178724086" watchObservedRunningTime="2026-10-30 02:00:06.603718996 +0000 UTC m=+2530.182006332" Oct 30 02:00:06.641413 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:06.641356 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888760-nbm5m" podStartSLOduration=1.851381238 podStartE2EDuration="6.641341677s" totalImagesPullingTime="4.789960439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:06.639206345 +0000 UTC m=+2530.217493683" watchObservedRunningTime="2026-10-30 02:00:06.641341677 +0000 UTC m=+2530.219629015" Oct 30 02:00:06.679748 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:06.679693 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888760-lfdmp" podStartSLOduration=2.044597843 podStartE2EDuration="6.679678978s" totalImagesPullingTime="4.635081135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:06.677732876 +0000 UTC m=+2530.256020214" watchObservedRunningTime="2026-10-30 02:00:06.679678978 +0000 UTC m=+2530.257966314" Oct 30 02:00:06.719571 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:06.719489 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f6e11e-29888760-sx6jd" podStartSLOduration=1.9483755569999999 podStartE2EDuration="6.719475449s" totalImagesPullingTime="4.771099892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:06.716363608 +0000 UTC m=+2530.294650945" watchObservedRunningTime="2026-10-30 02:00:06.719475449 +0000 UTC m=+2530.297762784" Oct 30 02:00:07.537981 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:07.537947 6221 generic.go:356] "Generic (PLEG): container finished" podID="76d7ecb2-375a-49ae-a766-db2b5405bfcd" containerID="26df5370feb3f95873d1657683d36f37321764be97eb77f3139a60579a086193" exitCode=60 Oct 30 02:00:07.538388 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:07.537989 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888760-jzwd6" event={"ID":"76d7ecb2-375a-49ae-a766-db2b5405bfcd","Type":"ContainerDied","Data":"26df5370feb3f95873d1657683d36f37321764be97eb77f3139a60579a086193"} Oct 30 02:00:07.538450 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:07.538396 6221 scope.go:122] "RemoveContainer" containerID="26df5370feb3f95873d1657683d36f37321764be97eb77f3139a60579a086193" Oct 30 02:00:07.539878 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:07.539852 6221 generic.go:356] "Generic (PLEG): container finished" podID="a451aa96-5403-409f-8a18-c1b2ea5a2231" containerID="7dbb8c59dfc9e7f92c8234abb8f1852a6efb60ad19857849f6ac4f012f5be402" exitCode=60 Oct 30 02:00:07.539996 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:07.539931 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888760-lfdmp" event={"ID":"a451aa96-5403-409f-8a18-c1b2ea5a2231","Type":"ContainerDied","Data":"7dbb8c59dfc9e7f92c8234abb8f1852a6efb60ad19857849f6ac4f012f5be402"} Oct 30 02:00:07.540146 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:07.540129 6221 scope.go:122] "RemoveContainer" containerID="7dbb8c59dfc9e7f92c8234abb8f1852a6efb60ad19857849f6ac4f012f5be402" Oct 30 02:00:07.541836 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:07.541812 6221 generic.go:356] "Generic (PLEG): container finished" podID="f1038f98-dfbd-4839-b07f-2cb9742f144c" containerID="87eb8d24cecc631c38cf67313c396739e84a4147abb5729ebdfd151b5054e7b6" exitCode=60 Oct 30 02:00:07.541965 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:07.541904 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888760-nbm5m" event={"ID":"f1038f98-dfbd-4839-b07f-2cb9742f144c","Type":"ContainerDied","Data":"87eb8d24cecc631c38cf67313c396739e84a4147abb5729ebdfd151b5054e7b6"} Oct 30 02:00:07.542111 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:07.542086 6221 scope.go:122] "RemoveContainer" containerID="87eb8d24cecc631c38cf67313c396739e84a4147abb5729ebdfd151b5054e7b6" Oct 30 02:00:07.543721 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:07.543631 6221 generic.go:356] "Generic (PLEG): container finished" podID="381d1132-541d-4397-91ab-285a4b873991" containerID="826a8c5954d0b5bdbb08cb55d0ddfd59c441cb2ad06b96e5313aaf8c5b777968" exitCode=60 Oct 30 02:00:07.543853 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:07.543774 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888760-qmg68" event={"ID":"381d1132-541d-4397-91ab-285a4b873991","Type":"ContainerDied","Data":"826a8c5954d0b5bdbb08cb55d0ddfd59c441cb2ad06b96e5313aaf8c5b777968"} Oct 30 02:00:07.544111 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:07.544090 6221 scope.go:122] "RemoveContainer" containerID="826a8c5954d0b5bdbb08cb55d0ddfd59c441cb2ad06b96e5313aaf8c5b777968" Oct 30 02:00:07.546098 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:07.546074 6221 generic.go:356] "Generic (PLEG): container finished" podID="108b6f30-ce76-41e2-a888-1d5601c4eeea" containerID="68bb88b705af3fb50adf62ae1194062ce7d0bc096b395a240ef4341c22074ad1" exitCode=60 Oct 30 02:00:07.546174 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:07.546113 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f6e11e-29888760-sx6jd" event={"ID":"108b6f30-ce76-41e2-a888-1d5601c4eeea","Type":"ContainerDied","Data":"68bb88b705af3fb50adf62ae1194062ce7d0bc096b395a240ef4341c22074ad1"} Oct 30 02:00:07.546397 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:07.546380 6221 scope.go:122] "RemoveContainer" containerID="68bb88b705af3fb50adf62ae1194062ce7d0bc096b395a240ef4341c22074ad1" Oct 30 02:00:08.555581 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:08.555526 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888760-qmg68" event={"ID":"381d1132-541d-4397-91ab-285a4b873991","Type":"ContainerStarted","Data":"fe91516c976d8aba560c38ccb63016a9acecf487729cb11edbff85714a17dc11"} Oct 30 02:00:08.557399 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:08.557370 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f6e11e-29888760-sx6jd" event={"ID":"108b6f30-ce76-41e2-a888-1d5601c4eeea","Type":"ContainerStarted","Data":"9b654004d26a8dc1ae21c83f04c38934cc7e7a2a098cbf1d9e737f8339106ec9"} Oct 30 02:00:08.559140 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:08.559115 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888760-jzwd6" event={"ID":"76d7ecb2-375a-49ae-a766-db2b5405bfcd","Type":"ContainerStarted","Data":"cea7bf4ed5ecc4c26143edd3bb8807d641198177c3514b2f58f0747cc655721f"} Oct 30 02:00:08.560928 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:08.560904 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888760-lfdmp" event={"ID":"a451aa96-5403-409f-8a18-c1b2ea5a2231","Type":"ContainerStarted","Data":"be44296a4e80ce21883af4c976d7005be6501968883467f2db541250b2b1e398"} Oct 30 02:00:08.562702 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:08.562681 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888760-nbm5m" event={"ID":"f1038f98-dfbd-4839-b07f-2cb9742f144c","Type":"ContainerStarted","Data":"313484949cfc62418d8d510fe2dc7b272c10c20cf536f3b551bf7cacc398a6b9"} Oct 30 02:00:09.572782 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:09.572747 6221 generic.go:356] "Generic (PLEG): container finished" podID="381d1132-541d-4397-91ab-285a4b873991" containerID="fe91516c976d8aba560c38ccb63016a9acecf487729cb11edbff85714a17dc11" exitCode=60 Oct 30 02:00:09.573214 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:09.572809 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888760-qmg68" event={"ID":"381d1132-541d-4397-91ab-285a4b873991","Type":"ContainerDied","Data":"fe91516c976d8aba560c38ccb63016a9acecf487729cb11edbff85714a17dc11"} Oct 30 02:00:09.573214 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:09.572876 6221 scope.go:122] "RemoveContainer" containerID="826a8c5954d0b5bdbb08cb55d0ddfd59c441cb2ad06b96e5313aaf8c5b777968" Oct 30 02:00:09.573214 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:09.573189 6221 scope.go:122] "RemoveContainer" containerID="fe91516c976d8aba560c38ccb63016a9acecf487729cb11edbff85714a17dc11" Oct 30 02:00:09.573498 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:00:09.573479 6221 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2e618f-29888760-qmg68_odh-ai-gateway-infra(381d1132-541d-4397-91ab-285a4b873991)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888760-qmg68" podUID="381d1132-541d-4397-91ab-285a4b873991" Oct 30 02:00:09.575431 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:09.575409 6221 generic.go:356] "Generic (PLEG): container finished" podID="108b6f30-ce76-41e2-a888-1d5601c4eeea" containerID="9b654004d26a8dc1ae21c83f04c38934cc7e7a2a098cbf1d9e737f8339106ec9" exitCode=60 Oct 30 02:00:09.575581 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:09.575496 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f6e11e-29888760-sx6jd" event={"ID":"108b6f30-ce76-41e2-a888-1d5601c4eeea","Type":"ContainerDied","Data":"9b654004d26a8dc1ae21c83f04c38934cc7e7a2a098cbf1d9e737f8339106ec9"} Oct 30 02:00:09.575742 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:09.575726 6221 scope.go:122] "RemoveContainer" containerID="9b654004d26a8dc1ae21c83f04c38934cc7e7a2a098cbf1d9e737f8339106ec9" Oct 30 02:00:09.575994 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:00:09.575976 6221 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-f6e11e-29888760-sx6jd_odh-ai-gateway-infra(108b6f30-ce76-41e2-a888-1d5601c4eeea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f6e11e-29888760-sx6jd" podUID="108b6f30-ce76-41e2-a888-1d5601c4eeea" Oct 30 02:00:09.577806 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:09.577779 6221 generic.go:356] "Generic (PLEG): container finished" podID="76d7ecb2-375a-49ae-a766-db2b5405bfcd" containerID="cea7bf4ed5ecc4c26143edd3bb8807d641198177c3514b2f58f0747cc655721f" exitCode=60 Oct 30 02:00:09.578242 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:09.578155 6221 scope.go:122] "RemoveContainer" containerID="cea7bf4ed5ecc4c26143edd3bb8807d641198177c3514b2f58f0747cc655721f" Oct 30 02:00:09.578367 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:09.578340 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888760-jzwd6" event={"ID":"76d7ecb2-375a-49ae-a766-db2b5405bfcd","Type":"ContainerDied","Data":"cea7bf4ed5ecc4c26143edd3bb8807d641198177c3514b2f58f0747cc655721f"} Oct 30 02:00:09.578506 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:00:09.578477 6221 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-cd8173-29888760-jzwd6_odh-ai-gateway-infra(76d7ecb2-375a-49ae-a766-db2b5405bfcd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888760-jzwd6" podUID="76d7ecb2-375a-49ae-a766-db2b5405bfcd" Oct 30 02:00:09.580242 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:09.580222 6221 generic.go:356] "Generic (PLEG): container finished" podID="a451aa96-5403-409f-8a18-c1b2ea5a2231" containerID="be44296a4e80ce21883af4c976d7005be6501968883467f2db541250b2b1e398" exitCode=60 Oct 30 02:00:09.580347 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:09.580285 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888760-lfdmp" event={"ID":"a451aa96-5403-409f-8a18-c1b2ea5a2231","Type":"ContainerDied","Data":"be44296a4e80ce21883af4c976d7005be6501968883467f2db541250b2b1e398"} Oct 30 02:00:09.580485 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:09.580471 6221 scope.go:122] "RemoveContainer" containerID="be44296a4e80ce21883af4c976d7005be6501968883467f2db541250b2b1e398" Oct 30 02:00:09.580764 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:00:09.580734 6221 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888760-lfdmp_odh-ai-gateway-infra(a451aa96-5403-409f-8a18-c1b2ea5a2231)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888760-lfdmp" podUID="a451aa96-5403-409f-8a18-c1b2ea5a2231" Oct 30 02:00:09.582282 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:09.582263 6221 generic.go:356] "Generic (PLEG): container finished" podID="f1038f98-dfbd-4839-b07f-2cb9742f144c" containerID="313484949cfc62418d8d510fe2dc7b272c10c20cf536f3b551bf7cacc398a6b9" exitCode=60 Oct 30 02:00:09.582380 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:09.582326 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888760-nbm5m" event={"ID":"f1038f98-dfbd-4839-b07f-2cb9742f144c","Type":"ContainerDied","Data":"313484949cfc62418d8d510fe2dc7b272c10c20cf536f3b551bf7cacc398a6b9"} Oct 30 02:00:09.582557 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:09.582518 6221 scope.go:122] "RemoveContainer" containerID="313484949cfc62418d8d510fe2dc7b272c10c20cf536f3b551bf7cacc398a6b9" Oct 30 02:00:09.582762 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:00:09.582740 6221 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888760-nbm5m_odh-ai-gateway-infra(f1038f98-dfbd-4839-b07f-2cb9742f144c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888760-nbm5m" podUID="f1038f98-dfbd-4839-b07f-2cb9742f144c" Oct 30 02:00:09.588835 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:09.588655 6221 scope.go:122] "RemoveContainer" containerID="68bb88b705af3fb50adf62ae1194062ce7d0bc096b395a240ef4341c22074ad1" Oct 30 02:00:09.604972 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:09.604952 6221 scope.go:122] "RemoveContainer" containerID="26df5370feb3f95873d1657683d36f37321764be97eb77f3139a60579a086193" Oct 30 02:00:09.618226 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:09.618202 6221 scope.go:122] "RemoveContainer" containerID="7dbb8c59dfc9e7f92c8234abb8f1852a6efb60ad19857849f6ac4f012f5be402" Oct 30 02:00:09.632926 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:09.632908 6221 scope.go:122] "RemoveContainer" containerID="87eb8d24cecc631c38cf67313c396739e84a4147abb5729ebdfd151b5054e7b6" Oct 30 02:00:10.591483 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:10.591454 6221 scope.go:122] "RemoveContainer" containerID="fe91516c976d8aba560c38ccb63016a9acecf487729cb11edbff85714a17dc11" Oct 30 02:00:10.591961 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:00:10.591794 6221 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2e618f-29888760-qmg68_odh-ai-gateway-infra(381d1132-541d-4397-91ab-285a4b873991)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888760-qmg68" podUID="381d1132-541d-4397-91ab-285a4b873991" Oct 30 02:00:10.592989 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:10.592966 6221 scope.go:122] "RemoveContainer" containerID="9b654004d26a8dc1ae21c83f04c38934cc7e7a2a098cbf1d9e737f8339106ec9" Oct 30 02:00:10.593248 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:00:10.593227 6221 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-f6e11e-29888760-sx6jd_odh-ai-gateway-infra(108b6f30-ce76-41e2-a888-1d5601c4eeea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f6e11e-29888760-sx6jd" podUID="108b6f30-ce76-41e2-a888-1d5601c4eeea" Oct 30 02:00:10.594455 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:10.594437 6221 scope.go:122] "RemoveContainer" containerID="cea7bf4ed5ecc4c26143edd3bb8807d641198177c3514b2f58f0747cc655721f" Oct 30 02:00:10.594739 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:00:10.594716 6221 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-cd8173-29888760-jzwd6_odh-ai-gateway-infra(76d7ecb2-375a-49ae-a766-db2b5405bfcd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888760-jzwd6" podUID="76d7ecb2-375a-49ae-a766-db2b5405bfcd" Oct 30 02:00:10.595913 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:10.595893 6221 scope.go:122] "RemoveContainer" containerID="be44296a4e80ce21883af4c976d7005be6501968883467f2db541250b2b1e398" Oct 30 02:00:10.596147 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:00:10.596128 6221 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888760-lfdmp_odh-ai-gateway-infra(a451aa96-5403-409f-8a18-c1b2ea5a2231)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888760-lfdmp" podUID="a451aa96-5403-409f-8a18-c1b2ea5a2231" Oct 30 02:00:10.597196 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:10.597181 6221 scope.go:122] "RemoveContainer" containerID="313484949cfc62418d8d510fe2dc7b272c10c20cf536f3b551bf7cacc398a6b9" Oct 30 02:00:10.597377 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:00:10.597360 6221 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888760-nbm5m_odh-ai-gateway-infra(f1038f98-dfbd-4839-b07f-2cb9742f144c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888760-nbm5m" podUID="f1038f98-dfbd-4839-b07f-2cb9742f144c" Oct 30 02:00:11.697656 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:00:11.697619 6221 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-093d23/agc-cred-723128d71ad3: secret "agc-cred-723128d71ad3" not found Oct 30 02:00:11.697656 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:00:11.697652 6221 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm: secret "agc-cred-723128d71ad3" not found Oct 30 02:00:11.698594 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:00:11.697724 6221 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07b7fc55-3b8b-4019-aae0-88231682d45e-provider-credentials podName:07b7fc55-3b8b-4019-aae0-88231682d45e nodeName:}" failed. No retries permitted until 2026-10-30 02:00:12.197705912 +0000 UTC m=+2535.775993243 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/07b7fc55-3b8b-4019-aae0-88231682d45e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm" (UID: "07b7fc55-3b8b-4019-aae0-88231682d45e") : secret "agc-cred-723128d71ad3" not found Oct 30 02:00:12.205468 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:00:12.205432 6221 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-093d23/agc-cred-723128d71ad3: secret "agc-cred-723128d71ad3" not found Oct 30 02:00:12.205468 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:00:12.205460 6221 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm: secret "agc-cred-723128d71ad3" not found Oct 30 02:00:12.205700 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:00:12.205526 6221 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07b7fc55-3b8b-4019-aae0-88231682d45e-provider-credentials podName:07b7fc55-3b8b-4019-aae0-88231682d45e nodeName:}" failed. No retries permitted until 2026-10-30 02:00:13.205507306 +0000 UTC m=+2536.783794661 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/07b7fc55-3b8b-4019-aae0-88231682d45e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm" (UID: "07b7fc55-3b8b-4019-aae0-88231682d45e") : secret "agc-cred-723128d71ad3" not found Oct 30 02:00:13.220103 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:00:13.220069 6221 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-093d23/agc-cred-723128d71ad3: secret "agc-cred-723128d71ad3" not found Oct 30 02:00:13.220103 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:00:13.220097 6221 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm: secret "agc-cred-723128d71ad3" not found Oct 30 02:00:13.220579 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:00:13.220152 6221 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07b7fc55-3b8b-4019-aae0-88231682d45e-provider-credentials podName:07b7fc55-3b8b-4019-aae0-88231682d45e nodeName:}" failed. No retries permitted until 2026-10-30 02:00:15.220136028 +0000 UTC m=+2538.798423341 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/07b7fc55-3b8b-4019-aae0-88231682d45e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm" (UID: "07b7fc55-3b8b-4019-aae0-88231682d45e") : secret "agc-cred-723128d71ad3" not found Oct 30 02:00:15.251352 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:00:15.251315 6221 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-093d23/agc-cred-723128d71ad3: secret "agc-cred-723128d71ad3" not found Oct 30 02:00:15.251352 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:00:15.251347 6221 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm: secret "agc-cred-723128d71ad3" not found Oct 30 02:00:15.251837 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:00:15.251409 6221 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07b7fc55-3b8b-4019-aae0-88231682d45e-provider-credentials podName:07b7fc55-3b8b-4019-aae0-88231682d45e nodeName:}" failed. No retries permitted until 2026-10-30 02:00:19.251390806 +0000 UTC m=+2542.829678126 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/07b7fc55-3b8b-4019-aae0-88231682d45e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm" (UID: "07b7fc55-3b8b-4019-aae0-88231682d45e") : secret "agc-cred-723128d71ad3" not found Oct 30 02:00:19.276639 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:00:19.276603 6221 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-093d23/agc-cred-723128d71ad3: secret "agc-cred-723128d71ad3" not found Oct 30 02:00:19.276639 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:00:19.276637 6221 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm: secret "agc-cred-723128d71ad3" not found Oct 30 02:00:19.277093 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:00:19.276714 6221 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07b7fc55-3b8b-4019-aae0-88231682d45e-provider-credentials podName:07b7fc55-3b8b-4019-aae0-88231682d45e nodeName:}" failed. No retries permitted until 2026-10-30 02:00:27.276698352 +0000 UTC m=+2550.854985666 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/07b7fc55-3b8b-4019-aae0-88231682d45e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm" (UID: "07b7fc55-3b8b-4019-aae0-88231682d45e") : secret "agc-cred-723128d71ad3" not found Oct 30 02:00:19.540802 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:19.540768 6221 scope.go:122] "RemoveContainer" containerID="9b654004d26a8dc1ae21c83f04c38934cc7e7a2a098cbf1d9e737f8339106ec9" Oct 30 02:00:19.540999 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:19.540972 6221 scope.go:122] "RemoveContainer" containerID="313484949cfc62418d8d510fe2dc7b272c10c20cf536f3b551bf7cacc398a6b9" Oct 30 02:00:19.541118 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:19.541098 6221 scope.go:122] "RemoveContainer" containerID="cea7bf4ed5ecc4c26143edd3bb8807d641198177c3514b2f58f0747cc655721f" Oct 30 02:00:20.541098 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:20.541067 6221 scope.go:122] "RemoveContainer" containerID="be44296a4e80ce21883af4c976d7005be6501968883467f2db541250b2b1e398" Oct 30 02:00:20.541540 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:20.541198 6221 scope.go:122] "RemoveContainer" containerID="fe91516c976d8aba560c38ccb63016a9acecf487729cb11edbff85714a17dc11" Oct 30 02:00:20.828018 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:20.825808 6221 generic.go:356] "Generic (PLEG): container finished" podID="108b6f30-ce76-41e2-a888-1d5601c4eeea" containerID="535f41b530f45b7c1c75685b18ebe414daf19fb6a9f4cb6ad4a17a4b5ed57067" exitCode=60 Oct 30 02:00:20.828018 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:20.825912 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f6e11e-29888760-sx6jd" event={"ID":"108b6f30-ce76-41e2-a888-1d5601c4eeea","Type":"ContainerDied","Data":"535f41b530f45b7c1c75685b18ebe414daf19fb6a9f4cb6ad4a17a4b5ed57067"} Oct 30 02:00:20.828018 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:20.825968 6221 scope.go:122] "RemoveContainer" containerID="9b654004d26a8dc1ae21c83f04c38934cc7e7a2a098cbf1d9e737f8339106ec9" Oct 30 02:00:20.828018 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:20.827849 6221 scope.go:122] "RemoveContainer" containerID="535f41b530f45b7c1c75685b18ebe414daf19fb6a9f4cb6ad4a17a4b5ed57067" Oct 30 02:00:20.828400 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:00:20.828270 6221 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-f6e11e-29888760-sx6jd_odh-ai-gateway-infra(108b6f30-ce76-41e2-a888-1d5601c4eeea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f6e11e-29888760-sx6jd" podUID="108b6f30-ce76-41e2-a888-1d5601c4eeea" Oct 30 02:00:20.853840 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:20.851212 6221 generic.go:356] "Generic (PLEG): container finished" podID="76d7ecb2-375a-49ae-a766-db2b5405bfcd" containerID="8f4a3627655f5beeabdc56d1c0a7df827770022c832e83bf56430e7ea7c9151a" exitCode=60 Oct 30 02:00:20.853840 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:20.851364 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888760-jzwd6" event={"ID":"76d7ecb2-375a-49ae-a766-db2b5405bfcd","Type":"ContainerDied","Data":"8f4a3627655f5beeabdc56d1c0a7df827770022c832e83bf56430e7ea7c9151a"} Oct 30 02:00:20.853840 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:20.851861 6221 scope.go:122] "RemoveContainer" containerID="8f4a3627655f5beeabdc56d1c0a7df827770022c832e83bf56430e7ea7c9151a" Oct 30 02:00:20.853840 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:00:20.852261 6221 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-cd8173-29888760-jzwd6_odh-ai-gateway-infra(76d7ecb2-375a-49ae-a766-db2b5405bfcd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888760-jzwd6" podUID="76d7ecb2-375a-49ae-a766-db2b5405bfcd" Oct 30 02:00:20.855568 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:20.854923 6221 generic.go:356] "Generic (PLEG): container finished" podID="f1038f98-dfbd-4839-b07f-2cb9742f144c" containerID="92fc0a526ae0256b8a6937d1088758cd8b3ec8a3655de208aad6c9724ba18773" exitCode=60 Oct 30 02:00:20.855568 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:20.855015 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888760-nbm5m" event={"ID":"f1038f98-dfbd-4839-b07f-2cb9742f144c","Type":"ContainerDied","Data":"92fc0a526ae0256b8a6937d1088758cd8b3ec8a3655de208aad6c9724ba18773"} Oct 30 02:00:20.855568 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:20.855411 6221 scope.go:122] "RemoveContainer" containerID="92fc0a526ae0256b8a6937d1088758cd8b3ec8a3655de208aad6c9724ba18773" Oct 30 02:00:20.856268 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:00:20.856205 6221 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888760-nbm5m_odh-ai-gateway-infra(f1038f98-dfbd-4839-b07f-2cb9742f144c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888760-nbm5m" podUID="f1038f98-dfbd-4839-b07f-2cb9742f144c" Oct 30 02:00:20.861067 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:20.859468 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76466xfk4"] Oct 30 02:00:20.865710 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:20.865636 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76466xfk4" Oct 30 02:00:20.870890 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:20.870868 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76466xfk4"] Oct 30 02:00:20.898498 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:20.898468 6221 scope.go:122] "RemoveContainer" containerID="cea7bf4ed5ecc4c26143edd3bb8807d641198177c3514b2f58f0747cc655721f" Oct 30 02:00:20.920772 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:20.920615 6221 scope.go:122] "RemoveContainer" containerID="313484949cfc62418d8d510fe2dc7b272c10c20cf536f3b551bf7cacc398a6b9" Oct 30 02:00:20.988545 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:20.988508 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm6f7\" (UniqueName: \"kubernetes.io/projected/f51c2d68-f0ff-4721-be1b-076584606b7e-kube-api-access-bm6f7\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-76466xfk4\" (UID: \"f51c2d68-f0ff-4721-be1b-076584606b7e\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76466xfk4" Oct 30 02:00:20.988733 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:20.988705 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f51c2d68-f0ff-4721-be1b-076584606b7e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-76466xfk4\" (UID: \"f51c2d68-f0ff-4721-be1b-076584606b7e\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76466xfk4" Oct 30 02:00:20.989025 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:20.989000 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f51c2d68-f0ff-4721-be1b-076584606b7e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-76466xfk4\" (UID: \"f51c2d68-f0ff-4721-be1b-076584606b7e\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76466xfk4" Oct 30 02:00:20.989223 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:20.989199 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f51c2d68-f0ff-4721-be1b-076584606b7e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-76466xfk4\" (UID: \"f51c2d68-f0ff-4721-be1b-076584606b7e\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76466xfk4" Oct 30 02:00:21.096339 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:21.091325 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f51c2d68-f0ff-4721-be1b-076584606b7e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-76466xfk4\" (UID: \"f51c2d68-f0ff-4721-be1b-076584606b7e\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76466xfk4" Oct 30 02:00:21.096339 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:21.091444 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm6f7\" (UniqueName: \"kubernetes.io/projected/f51c2d68-f0ff-4721-be1b-076584606b7e-kube-api-access-bm6f7\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-76466xfk4\" (UID: \"f51c2d68-f0ff-4721-be1b-076584606b7e\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76466xfk4" Oct 30 02:00:21.096339 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:21.091689 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f51c2d68-f0ff-4721-be1b-076584606b7e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-76466xfk4\" (UID: \"f51c2d68-f0ff-4721-be1b-076584606b7e\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76466xfk4" Oct 30 02:00:21.096339 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:21.092157 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f51c2d68-f0ff-4721-be1b-076584606b7e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-76466xfk4\" (UID: \"f51c2d68-f0ff-4721-be1b-076584606b7e\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76466xfk4" Oct 30 02:00:21.096751 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:21.096725 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f51c2d68-f0ff-4721-be1b-076584606b7e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-76466xfk4\" (UID: \"f51c2d68-f0ff-4721-be1b-076584606b7e\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76466xfk4" Oct 30 02:00:21.097433 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:21.097413 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f51c2d68-f0ff-4721-be1b-076584606b7e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-76466xfk4\" (UID: \"f51c2d68-f0ff-4721-be1b-076584606b7e\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76466xfk4" Oct 30 02:00:21.111278 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:21.111232 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm6f7\" (UniqueName: \"kubernetes.io/projected/f51c2d68-f0ff-4721-be1b-076584606b7e-kube-api-access-bm6f7\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-76466xfk4\" (UID: \"f51c2d68-f0ff-4721-be1b-076584606b7e\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76466xfk4" Oct 30 02:00:21.115278 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:21.115253 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f51c2d68-f0ff-4721-be1b-076584606b7e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-093d23-76466xfk4\" (UID: \"f51c2d68-f0ff-4721-be1b-076584606b7e\") " pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76466xfk4" Oct 30 02:00:21.222029 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:21.222000 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76466xfk4" Oct 30 02:00:21.502630 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:00:21.502577 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 02:00:21.556242 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:21.556221 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76466xfk4"] Oct 30 02:00:21.558317 ip-10-0-135-65 kubenswrapper[6221]: W1030 02:00:21.558292 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf51c2d68_f0ff_4721_be1b_076584606b7e.slice/crio-5e4a3138ae00448011f0919ab6d8c652a10670b4f9579231cdc4d228849b6093 WatchSource:0}: Error finding container 5e4a3138ae00448011f0919ab6d8c652a10670b4f9579231cdc4d228849b6093: Status 404 returned error can't find the container with id 5e4a3138ae00448011f0919ab6d8c652a10670b4f9579231cdc4d228849b6093 Oct 30 02:00:21.864029 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:21.863943 6221 generic.go:356] "Generic (PLEG): container finished" podID="a451aa96-5403-409f-8a18-c1b2ea5a2231" containerID="77ba0baff2ab74edabe163afb0c8f3b344ba9677e58952d1daa9ed9f10315861" exitCode=60 Oct 30 02:00:21.864029 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:21.864012 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888760-lfdmp" event={"ID":"a451aa96-5403-409f-8a18-c1b2ea5a2231","Type":"ContainerDied","Data":"77ba0baff2ab74edabe163afb0c8f3b344ba9677e58952d1daa9ed9f10315861"} Oct 30 02:00:21.864256 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:21.864089 6221 scope.go:122] "RemoveContainer" containerID="be44296a4e80ce21883af4c976d7005be6501968883467f2db541250b2b1e398" Oct 30 02:00:21.864519 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:21.864502 6221 scope.go:122] "RemoveContainer" containerID="77ba0baff2ab74edabe163afb0c8f3b344ba9677e58952d1daa9ed9f10315861" Oct 30 02:00:21.864926 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:00:21.864900 6221 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888760-lfdmp_odh-ai-gateway-infra(a451aa96-5403-409f-8a18-c1b2ea5a2231)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888760-lfdmp" podUID="a451aa96-5403-409f-8a18-c1b2ea5a2231" Oct 30 02:00:21.868232 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:21.868210 6221 generic.go:356] "Generic (PLEG): container finished" podID="381d1132-541d-4397-91ab-285a4b873991" containerID="8dd28c6bc8dd3ce68ec18743e62a2890f0811cafa0fd74e41b6f262f3a670075" exitCode=60 Oct 30 02:00:21.868329 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:21.868285 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888760-qmg68" event={"ID":"381d1132-541d-4397-91ab-285a4b873991","Type":"ContainerDied","Data":"8dd28c6bc8dd3ce68ec18743e62a2890f0811cafa0fd74e41b6f262f3a670075"} Oct 30 02:00:21.868730 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:21.868710 6221 scope.go:122] "RemoveContainer" containerID="8dd28c6bc8dd3ce68ec18743e62a2890f0811cafa0fd74e41b6f262f3a670075" Oct 30 02:00:21.869087 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:00:21.869069 6221 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2e618f-29888760-qmg68_odh-ai-gateway-infra(381d1132-541d-4397-91ab-285a4b873991)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888760-qmg68" podUID="381d1132-541d-4397-91ab-285a4b873991" Oct 30 02:00:21.871773 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:21.871747 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76466xfk4" event={"ID":"f51c2d68-f0ff-4721-be1b-076584606b7e","Type":"ContainerStarted","Data":"e6ef40e0cadc42602eb3ab9fee34501d925b04b58ff8d542e5a3d796150c1356"} Oct 30 02:00:21.871874 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:21.871802 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76466xfk4" event={"ID":"f51c2d68-f0ff-4721-be1b-076584606b7e","Type":"ContainerStarted","Data":"5e4a3138ae00448011f0919ab6d8c652a10670b4f9579231cdc4d228849b6093"} Oct 30 02:00:21.871874 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:21.871833 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76466xfk4" Oct 30 02:00:21.879012 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:21.878982 6221 scope.go:122] "RemoveContainer" containerID="fe91516c976d8aba560c38ccb63016a9acecf487729cb11edbff85714a17dc11" Oct 30 02:00:21.897084 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:21.897059 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f6e11e-29888760-sx6jd"] Oct 30 02:00:21.938451 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:21.938420 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888760-jzwd6"] Oct 30 02:00:21.954407 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:21.954352 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76466xfk4" podStartSLOduration=1.954334045 podStartE2EDuration="1.954334045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:00:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:00:21.949051321 +0000 UTC m=+2545.527338657" watchObservedRunningTime="2026-10-30 02:00:21.954334045 +0000 UTC m=+2545.532621382" Oct 30 02:00:21.993757 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:21.993699 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888760-nbm5m"] Oct 30 02:00:22.096059 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:22.096024 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f6e11e-29888760-sx6jd" Oct 30 02:00:22.176971 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:22.176938 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888760-jzwd6" Oct 30 02:00:22.211823 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:22.211799 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888760-nbm5m" Oct 30 02:00:22.225286 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:22.225252 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/108b6f30-ce76-41e2-a888-1d5601c4eeea-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/108b6f30-ce76-41e2-a888-1d5601c4eeea-maas-api-service-ca\") pod \"108b6f30-ce76-41e2-a888-1d5601c4eeea\" (UID: \"108b6f30-ce76-41e2-a888-1d5601c4eeea\") " Oct 30 02:00:22.225447 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:22.225400 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/108b6f30-ce76-41e2-a888-1d5601c4eeea-kube-api-access-rn5d4\" (UniqueName: \"kubernetes.io/projected/108b6f30-ce76-41e2-a888-1d5601c4eeea-kube-api-access-rn5d4\") pod \"108b6f30-ce76-41e2-a888-1d5601c4eeea\" (UID: \"108b6f30-ce76-41e2-a888-1d5601c4eeea\") " Oct 30 02:00:22.225628 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:22.225608 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/108b6f30-ce76-41e2-a888-1d5601c4eeea-maas-api-service-ca" pod "108b6f30-ce76-41e2-a888-1d5601c4eeea" (UID: "108b6f30-ce76-41e2-a888-1d5601c4eeea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:22.226070 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:22.226048 6221 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/108b6f30-ce76-41e2-a888-1d5601c4eeea-maas-api-service-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:00:22.227717 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:22.227694 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/108b6f30-ce76-41e2-a888-1d5601c4eeea-kube-api-access-rn5d4" pod "108b6f30-ce76-41e2-a888-1d5601c4eeea" (UID: "108b6f30-ce76-41e2-a888-1d5601c4eeea"). InnerVolumeSpecName "kube-api-access-rn5d4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:22.327191 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:22.327161 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76d7ecb2-375a-49ae-a766-db2b5405bfcd-kube-api-access-9xqrj\" (UniqueName: \"kubernetes.io/projected/76d7ecb2-375a-49ae-a766-db2b5405bfcd-kube-api-access-9xqrj\") pod \"76d7ecb2-375a-49ae-a766-db2b5405bfcd\" (UID: \"76d7ecb2-375a-49ae-a766-db2b5405bfcd\") " Oct 30 02:00:22.327359 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:22.327207 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1038f98-dfbd-4839-b07f-2cb9742f144c-kube-api-access-xgg45\" (UniqueName: \"kubernetes.io/projected/f1038f98-dfbd-4839-b07f-2cb9742f144c-kube-api-access-xgg45\") pod \"f1038f98-dfbd-4839-b07f-2cb9742f144c\" (UID: \"f1038f98-dfbd-4839-b07f-2cb9742f144c\") " Oct 30 02:00:22.327359 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:22.327305 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1038f98-dfbd-4839-b07f-2cb9742f144c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1038f98-dfbd-4839-b07f-2cb9742f144c-maas-api-service-ca\") pod \"f1038f98-dfbd-4839-b07f-2cb9742f144c\" (UID: \"f1038f98-dfbd-4839-b07f-2cb9742f144c\") " Oct 30 02:00:22.327581 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:22.327560 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76d7ecb2-375a-49ae-a766-db2b5405bfcd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76d7ecb2-375a-49ae-a766-db2b5405bfcd-maas-api-service-ca\") pod \"76d7ecb2-375a-49ae-a766-db2b5405bfcd\" (UID: \"76d7ecb2-375a-49ae-a766-db2b5405bfcd\") " Oct 30 02:00:22.327768 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:22.327712 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1038f98-dfbd-4839-b07f-2cb9742f144c-maas-api-service-ca" pod "f1038f98-dfbd-4839-b07f-2cb9742f144c" (UID: "f1038f98-dfbd-4839-b07f-2cb9742f144c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:22.327882 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:22.327861 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76d7ecb2-375a-49ae-a766-db2b5405bfcd-maas-api-service-ca" pod "76d7ecb2-375a-49ae-a766-db2b5405bfcd" (UID: "76d7ecb2-375a-49ae-a766-db2b5405bfcd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:22.328167 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:22.328149 6221 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1038f98-dfbd-4839-b07f-2cb9742f144c-maas-api-service-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:00:22.328236 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:22.328168 6221 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76d7ecb2-375a-49ae-a766-db2b5405bfcd-maas-api-service-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:00:22.328236 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:22.328177 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn5d4\" (UniqueName: \"kubernetes.io/projected/108b6f30-ce76-41e2-a888-1d5601c4eeea-kube-api-access-rn5d4\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:00:22.329472 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:22.329445 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1038f98-dfbd-4839-b07f-2cb9742f144c-kube-api-access-xgg45" pod "f1038f98-dfbd-4839-b07f-2cb9742f144c" (UID: "f1038f98-dfbd-4839-b07f-2cb9742f144c"). InnerVolumeSpecName "kube-api-access-xgg45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:22.329585 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:22.329475 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76d7ecb2-375a-49ae-a766-db2b5405bfcd-kube-api-access-9xqrj" pod "76d7ecb2-375a-49ae-a766-db2b5405bfcd" (UID: "76d7ecb2-375a-49ae-a766-db2b5405bfcd"). InnerVolumeSpecName "kube-api-access-9xqrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:22.429987 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:22.429917 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xqrj\" (UniqueName: \"kubernetes.io/projected/76d7ecb2-375a-49ae-a766-db2b5405bfcd-kube-api-access-9xqrj\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:00:22.429987 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:22.429947 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgg45\" (UniqueName: \"kubernetes.io/projected/f1038f98-dfbd-4839-b07f-2cb9742f144c-kube-api-access-xgg45\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:00:22.883355 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:22.883323 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f6e11e-29888760-sx6jd" Oct 30 02:00:22.883908 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:22.883328 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f6e11e-29888760-sx6jd" event={"ID":"108b6f30-ce76-41e2-a888-1d5601c4eeea","Type":"ContainerDied","Data":"3c390d3185b8b47b0b3725d8a4dcddc508ba69125d4f690ef053bf3a10ef37af"} Oct 30 02:00:22.883908 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:22.883454 6221 scope.go:122] "RemoveContainer" containerID="535f41b530f45b7c1c75685b18ebe414daf19fb6a9f4cb6ad4a17a4b5ed57067" Oct 30 02:00:22.884854 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:22.884835 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888760-jzwd6" Oct 30 02:00:22.884910 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:22.884859 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888760-jzwd6" event={"ID":"76d7ecb2-375a-49ae-a766-db2b5405bfcd","Type":"ContainerDied","Data":"dbede8a07d209de4dcc03842d216627d59ca903a4355aa92beb7b1522d77e0f7"} Oct 30 02:00:22.888420 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:22.888403 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888760-nbm5m" Oct 30 02:00:22.888548 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:22.888464 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888760-nbm5m" event={"ID":"f1038f98-dfbd-4839-b07f-2cb9742f144c","Type":"ContainerDied","Data":"08c2fde73b68bd4feb091d550518745f6378e7e07bff01a128d31c0a52549c0e"} Oct 30 02:00:22.895427 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:22.895410 6221 scope.go:122] "RemoveContainer" containerID="8f4a3627655f5beeabdc56d1c0a7df827770022c832e83bf56430e7ea7c9151a" Oct 30 02:00:22.904622 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:22.904603 6221 scope.go:122] "RemoveContainer" containerID="92fc0a526ae0256b8a6937d1088758cd8b3ec8a3655de208aad6c9724ba18773" Oct 30 02:00:22.922143 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:22.922119 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888760-lfdmp"] Oct 30 02:00:22.945020 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:22.944983 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888760-jzwd6"] Oct 30 02:00:22.957430 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:22.957403 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-cd8173-29888760-jzwd6"] Oct 30 02:00:23.000965 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:23.000929 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888760-nbm5m"] Oct 30 02:00:23.008969 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:23.008941 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-b6b7fc-29888760-nbm5m"] Oct 30 02:00:23.046014 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:23.045983 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f6e11e-29888760-sx6jd"] Oct 30 02:00:23.050604 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:23.050572 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888760-qmg68"] Oct 30 02:00:23.054374 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:23.054351 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-f6e11e-29888760-sx6jd"] Oct 30 02:00:23.121244 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:23.121210 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888760-lfdmp" Oct 30 02:00:23.242852 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:23.242828 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a451aa96-5403-409f-8a18-c1b2ea5a2231-kube-api-access-mb7ts\" (UniqueName: \"kubernetes.io/projected/a451aa96-5403-409f-8a18-c1b2ea5a2231-kube-api-access-mb7ts\") pod \"a451aa96-5403-409f-8a18-c1b2ea5a2231\" (UID: \"a451aa96-5403-409f-8a18-c1b2ea5a2231\") " Oct 30 02:00:23.242987 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:23.242876 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a451aa96-5403-409f-8a18-c1b2ea5a2231-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a451aa96-5403-409f-8a18-c1b2ea5a2231-maas-api-service-ca\") pod \"a451aa96-5403-409f-8a18-c1b2ea5a2231\" (UID: \"a451aa96-5403-409f-8a18-c1b2ea5a2231\") " Oct 30 02:00:23.243250 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:23.243228 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a451aa96-5403-409f-8a18-c1b2ea5a2231-maas-api-service-ca" pod "a451aa96-5403-409f-8a18-c1b2ea5a2231" (UID: "a451aa96-5403-409f-8a18-c1b2ea5a2231"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:23.243454 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:23.243436 6221 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a451aa96-5403-409f-8a18-c1b2ea5a2231-maas-api-service-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:00:23.244949 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:23.244918 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a451aa96-5403-409f-8a18-c1b2ea5a2231-kube-api-access-mb7ts" pod "a451aa96-5403-409f-8a18-c1b2ea5a2231" (UID: "a451aa96-5403-409f-8a18-c1b2ea5a2231"). InnerVolumeSpecName "kube-api-access-mb7ts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:23.252511 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:23.252490 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888760-qmg68" Oct 30 02:00:23.344818 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:23.344785 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/381d1132-541d-4397-91ab-285a4b873991-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/381d1132-541d-4397-91ab-285a4b873991-maas-api-service-ca\") pod \"381d1132-541d-4397-91ab-285a4b873991\" (UID: \"381d1132-541d-4397-91ab-285a4b873991\") " Oct 30 02:00:23.344990 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:23.344856 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/381d1132-541d-4397-91ab-285a4b873991-kube-api-access-nlfnp\" (UniqueName: \"kubernetes.io/projected/381d1132-541d-4397-91ab-285a4b873991-kube-api-access-nlfnp\") pod \"381d1132-541d-4397-91ab-285a4b873991\" (UID: \"381d1132-541d-4397-91ab-285a4b873991\") " Oct 30 02:00:23.345221 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:23.345196 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/381d1132-541d-4397-91ab-285a4b873991-maas-api-service-ca" pod "381d1132-541d-4397-91ab-285a4b873991" (UID: "381d1132-541d-4397-91ab-285a4b873991"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:23.345385 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:23.345370 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb7ts\" (UniqueName: \"kubernetes.io/projected/a451aa96-5403-409f-8a18-c1b2ea5a2231-kube-api-access-mb7ts\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:00:23.345429 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:23.345389 6221 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/381d1132-541d-4397-91ab-285a4b873991-maas-api-service-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:00:23.347065 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:23.347044 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/381d1132-541d-4397-91ab-285a4b873991-kube-api-access-nlfnp" pod "381d1132-541d-4397-91ab-285a4b873991" (UID: "381d1132-541d-4397-91ab-285a4b873991"). InnerVolumeSpecName "kube-api-access-nlfnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:23.447139 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:23.447100 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlfnp\" (UniqueName: \"kubernetes.io/projected/381d1132-541d-4397-91ab-285a4b873991-kube-api-access-nlfnp\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:00:23.899291 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:23.899189 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888760-qmg68" event={"ID":"381d1132-541d-4397-91ab-285a4b873991","Type":"ContainerDied","Data":"fe6e7780638c77753e930699be93240ca42af3af64db626a21fea443920484ed"} Oct 30 02:00:23.899291 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:23.899212 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888760-qmg68" Oct 30 02:00:23.899828 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:23.899301 6221 scope.go:122] "RemoveContainer" containerID="8dd28c6bc8dd3ce68ec18743e62a2890f0811cafa0fd74e41b6f262f3a670075" Oct 30 02:00:23.903520 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:23.903490 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888760-lfdmp" event={"ID":"a451aa96-5403-409f-8a18-c1b2ea5a2231","Type":"ContainerDied","Data":"3e35eafb228efde72afec357a7d639f918aa366a49562c8718e51bb231391d18"} Oct 30 02:00:23.903696 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:23.903553 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888760-lfdmp" Oct 30 02:00:23.917147 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:23.917086 6221 scope.go:122] "RemoveContainer" containerID="77ba0baff2ab74edabe163afb0c8f3b344ba9677e58952d1daa9ed9f10315861" Oct 30 02:00:23.969613 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:23.969577 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888760-qmg68"] Oct 30 02:00:23.973593 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:23.973568 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888760-qmg68"] Oct 30 02:00:24.019008 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:24.018976 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888760-lfdmp"] Oct 30 02:00:24.044018 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:24.043987 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888760-lfdmp"] Oct 30 02:00:24.548768 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:24.548733 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="108b6f30-ce76-41e2-a888-1d5601c4eeea" path="/var/lib/kubelet/pods/108b6f30-ce76-41e2-a888-1d5601c4eeea/volumes" Oct 30 02:00:24.549249 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:24.549227 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="381d1132-541d-4397-91ab-285a4b873991" path="/var/lib/kubelet/pods/381d1132-541d-4397-91ab-285a4b873991/volumes" Oct 30 02:00:24.549665 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:24.549652 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76d7ecb2-375a-49ae-a766-db2b5405bfcd" path="/var/lib/kubelet/pods/76d7ecb2-375a-49ae-a766-db2b5405bfcd/volumes" Oct 30 02:00:24.550357 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:24.550342 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a451aa96-5403-409f-8a18-c1b2ea5a2231" path="/var/lib/kubelet/pods/a451aa96-5403-409f-8a18-c1b2ea5a2231/volumes" Oct 30 02:00:24.550775 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:24.550762 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1038f98-dfbd-4839-b07f-2cb9742f144c" path="/var/lib/kubelet/pods/f1038f98-dfbd-4839-b07f-2cb9742f144c/volumes" Oct 30 02:00:27.304991 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:00:27.304953 6221 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-093d23/agc-cred-1eeb94896bc0: secret "agc-cred-1eeb94896bc0" not found Oct 30 02:00:27.304991 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:00:27.304988 6221 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-093d23/agc-cred-723128d71ad3: secret "agc-cred-723128d71ad3" not found Oct 30 02:00:27.305560 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:00:27.305003 6221 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm: [secret "agc-cred-1eeb94896bc0" not found, secret "agc-cred-723128d71ad3" not found] Oct 30 02:00:27.305560 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:00:27.305066 6221 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/07b7fc55-3b8b-4019-aae0-88231682d45e-provider-credentials podName:07b7fc55-3b8b-4019-aae0-88231682d45e nodeName:}" failed. No retries permitted until 2026-10-30 02:00:43.305052214 +0000 UTC m=+2566.883339541 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/07b7fc55-3b8b-4019-aae0-88231682d45e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm" (UID: "07b7fc55-3b8b-4019-aae0-88231682d45e") : [secret "agc-cred-1eeb94896bc0" not found, secret "agc-cred-723128d71ad3" not found] Oct 30 02:00:32.380711 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:32.380677 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76466xfk4"] Oct 30 02:00:32.384685 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:32.384662 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76466xfk4" Oct 30 02:00:32.393604 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:32.393572 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm"] Oct 30 02:00:37.381691 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:37.381648 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76466xfk4" podUID="f51c2d68-f0ff-4721-be1b-076584606b7e" containerName="payload-processing" containerID="cri-o://e6ef40e0cadc42602eb3ab9fee34501d925b04b58ff8d542e5a3d796150c1356" gracePeriod=25 Oct 30 02:00:37.393909 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:37.393880 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm" podUID="07b7fc55-3b8b-4019-aae0-88231682d45e" containerName="payload-processing" containerID="cri-o://4e16b7ee26d0985febad68a2dccd8fcd9e71783ee0f46fc55897f64ef1cd2426" gracePeriod=25 Oct 30 02:00:37.685790 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:37.685762 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76466xfk4" Oct 30 02:00:37.707384 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:37.707352 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm" Oct 30 02:00:37.796601 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:37.796526 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07b7fc55-3b8b-4019-aae0-88231682d45e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07b7fc55-3b8b-4019-aae0-88231682d45e-plugins-config-volume\") pod \"07b7fc55-3b8b-4019-aae0-88231682d45e\" (UID: \"07b7fc55-3b8b-4019-aae0-88231682d45e\") " Oct 30 02:00:37.796800 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:37.796644 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07b7fc55-3b8b-4019-aae0-88231682d45e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/07b7fc55-3b8b-4019-aae0-88231682d45e-provider-credentials\") pod \"07b7fc55-3b8b-4019-aae0-88231682d45e\" (UID: \"07b7fc55-3b8b-4019-aae0-88231682d45e\") " Oct 30 02:00:37.796800 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:37.796794 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f51c2d68-f0ff-4721-be1b-076584606b7e-kube-api-access-bm6f7\" (UniqueName: \"kubernetes.io/projected/f51c2d68-f0ff-4721-be1b-076584606b7e-kube-api-access-bm6f7\") pod \"f51c2d68-f0ff-4721-be1b-076584606b7e\" (UID: \"f51c2d68-f0ff-4721-be1b-076584606b7e\") " Oct 30 02:00:37.796931 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:37.796875 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07b7fc55-3b8b-4019-aae0-88231682d45e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07b7fc55-3b8b-4019-aae0-88231682d45e-routing-overlay-volume\") pod \"07b7fc55-3b8b-4019-aae0-88231682d45e\" (UID: \"07b7fc55-3b8b-4019-aae0-88231682d45e\") " Oct 30 02:00:37.796931 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:37.796917 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f51c2d68-f0ff-4721-be1b-076584606b7e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f51c2d68-f0ff-4721-be1b-076584606b7e-plugins-config-volume\") pod \"f51c2d68-f0ff-4721-be1b-076584606b7e\" (UID: \"f51c2d68-f0ff-4721-be1b-076584606b7e\") " Oct 30 02:00:37.797050 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:37.796963 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f51c2d68-f0ff-4721-be1b-076584606b7e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f51c2d68-f0ff-4721-be1b-076584606b7e-provider-credentials\") pod \"f51c2d68-f0ff-4721-be1b-076584606b7e\" (UID: \"f51c2d68-f0ff-4721-be1b-076584606b7e\") " Oct 30 02:00:37.797109 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:37.797048 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f51c2d68-f0ff-4721-be1b-076584606b7e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f51c2d68-f0ff-4721-be1b-076584606b7e-routing-overlay-volume\") pod \"f51c2d68-f0ff-4721-be1b-076584606b7e\" (UID: \"f51c2d68-f0ff-4721-be1b-076584606b7e\") " Oct 30 02:00:37.797166 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:37.797112 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07b7fc55-3b8b-4019-aae0-88231682d45e-kube-api-access-gfx55\" (UniqueName: \"kubernetes.io/projected/07b7fc55-3b8b-4019-aae0-88231682d45e-kube-api-access-gfx55\") pod \"07b7fc55-3b8b-4019-aae0-88231682d45e\" (UID: \"07b7fc55-3b8b-4019-aae0-88231682d45e\") " Oct 30 02:00:37.797222 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:37.797164 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07b7fc55-3b8b-4019-aae0-88231682d45e-plugins-config-volume" pod "07b7fc55-3b8b-4019-aae0-88231682d45e" (UID: "07b7fc55-3b8b-4019-aae0-88231682d45e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:37.797547 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:37.797509 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07b7fc55-3b8b-4019-aae0-88231682d45e-routing-overlay-volume" pod "07b7fc55-3b8b-4019-aae0-88231682d45e" (UID: "07b7fc55-3b8b-4019-aae0-88231682d45e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:37.797662 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:37.797637 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f51c2d68-f0ff-4721-be1b-076584606b7e-plugins-config-volume" pod "f51c2d68-f0ff-4721-be1b-076584606b7e" (UID: "f51c2d68-f0ff-4721-be1b-076584606b7e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:37.797947 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:37.797921 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f51c2d68-f0ff-4721-be1b-076584606b7e-routing-overlay-volume" pod "f51c2d68-f0ff-4721-be1b-076584606b7e" (UID: "f51c2d68-f0ff-4721-be1b-076584606b7e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:00:37.798036 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:37.797980 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07b7fc55-3b8b-4019-aae0-88231682d45e-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:00:37.798036 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:37.798003 6221 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/07b7fc55-3b8b-4019-aae0-88231682d45e-routing-overlay-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:00:37.798036 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:37.798018 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f51c2d68-f0ff-4721-be1b-076584606b7e-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:00:37.799657 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:37.799626 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07b7fc55-3b8b-4019-aae0-88231682d45e-provider-credentials" pod "07b7fc55-3b8b-4019-aae0-88231682d45e" (UID: "07b7fc55-3b8b-4019-aae0-88231682d45e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:37.799923 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:37.799901 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f51c2d68-f0ff-4721-be1b-076584606b7e-kube-api-access-bm6f7" pod "f51c2d68-f0ff-4721-be1b-076584606b7e" (UID: "f51c2d68-f0ff-4721-be1b-076584606b7e"). InnerVolumeSpecName "kube-api-access-bm6f7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:37.800168 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:37.800144 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07b7fc55-3b8b-4019-aae0-88231682d45e-kube-api-access-gfx55" pod "07b7fc55-3b8b-4019-aae0-88231682d45e" (UID: "07b7fc55-3b8b-4019-aae0-88231682d45e"). InnerVolumeSpecName "kube-api-access-gfx55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:37.800281 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:37.800260 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f51c2d68-f0ff-4721-be1b-076584606b7e-provider-credentials" pod "f51c2d68-f0ff-4721-be1b-076584606b7e" (UID: "f51c2d68-f0ff-4721-be1b-076584606b7e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:00:37.899317 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:37.899226 6221 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f51c2d68-f0ff-4721-be1b-076584606b7e-provider-credentials\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:00:37.899317 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:37.899262 6221 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f51c2d68-f0ff-4721-be1b-076584606b7e-routing-overlay-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:00:37.899317 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:37.899272 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfx55\" (UniqueName: \"kubernetes.io/projected/07b7fc55-3b8b-4019-aae0-88231682d45e-kube-api-access-gfx55\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:00:37.899317 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:37.899285 6221 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/07b7fc55-3b8b-4019-aae0-88231682d45e-provider-credentials\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:00:37.899317 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:37.899298 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm6f7\" (UniqueName: \"kubernetes.io/projected/f51c2d68-f0ff-4721-be1b-076584606b7e-kube-api-access-bm6f7\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:00:38.010094 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:38.010056 6221 generic.go:356] "Generic (PLEG): container finished" podID="07b7fc55-3b8b-4019-aae0-88231682d45e" containerID="4e16b7ee26d0985febad68a2dccd8fcd9e71783ee0f46fc55897f64ef1cd2426" exitCode=0 Oct 30 02:00:38.010316 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:38.010105 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm" Oct 30 02:00:38.010316 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:38.010154 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm" event={"ID":"07b7fc55-3b8b-4019-aae0-88231682d45e","Type":"ContainerDied","Data":"4e16b7ee26d0985febad68a2dccd8fcd9e71783ee0f46fc55897f64ef1cd2426"} Oct 30 02:00:38.010316 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:38.010242 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm" event={"ID":"07b7fc55-3b8b-4019-aae0-88231682d45e","Type":"ContainerDied","Data":"abbba351d598363ae6ca391b0592334a85e4569335dbae30d849cd9d4d18e26e"} Oct 30 02:00:38.010316 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:38.010273 6221 scope.go:122] "RemoveContainer" containerID="4e16b7ee26d0985febad68a2dccd8fcd9e71783ee0f46fc55897f64ef1cd2426" Oct 30 02:00:38.011892 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:38.011850 6221 generic.go:356] "Generic (PLEG): container finished" podID="f51c2d68-f0ff-4721-be1b-076584606b7e" containerID="e6ef40e0cadc42602eb3ab9fee34501d925b04b58ff8d542e5a3d796150c1356" exitCode=0 Oct 30 02:00:38.012079 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:38.011913 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76466xfk4" Oct 30 02:00:38.012079 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:38.011955 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76466xfk4" event={"ID":"f51c2d68-f0ff-4721-be1b-076584606b7e","Type":"ContainerDied","Data":"e6ef40e0cadc42602eb3ab9fee34501d925b04b58ff8d542e5a3d796150c1356"} Oct 30 02:00:38.012079 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:38.011998 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76466xfk4" event={"ID":"f51c2d68-f0ff-4721-be1b-076584606b7e","Type":"ContainerDied","Data":"5e4a3138ae00448011f0919ab6d8c652a10670b4f9579231cdc4d228849b6093"} Oct 30 02:00:38.022520 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:38.022498 6221 scope.go:122] "RemoveContainer" containerID="4e16b7ee26d0985febad68a2dccd8fcd9e71783ee0f46fc55897f64ef1cd2426" Oct 30 02:00:38.022873 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:00:38.022845 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e16b7ee26d0985febad68a2dccd8fcd9e71783ee0f46fc55897f64ef1cd2426\": container with ID starting with 4e16b7ee26d0985febad68a2dccd8fcd9e71783ee0f46fc55897f64ef1cd2426 not found: ID does not exist" containerID="4e16b7ee26d0985febad68a2dccd8fcd9e71783ee0f46fc55897f64ef1cd2426" Oct 30 02:00:38.022959 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:38.022883 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e16b7ee26d0985febad68a2dccd8fcd9e71783ee0f46fc55897f64ef1cd2426"} err="failed to get container status \"4e16b7ee26d0985febad68a2dccd8fcd9e71783ee0f46fc55897f64ef1cd2426\": rpc error: code = NotFound desc = could not find container \"4e16b7ee26d0985febad68a2dccd8fcd9e71783ee0f46fc55897f64ef1cd2426\": container with ID starting with 4e16b7ee26d0985febad68a2dccd8fcd9e71783ee0f46fc55897f64ef1cd2426 not found: ID does not exist" Oct 30 02:00:38.022959 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:38.022901 6221 scope.go:122] "RemoveContainer" containerID="e6ef40e0cadc42602eb3ab9fee34501d925b04b58ff8d542e5a3d796150c1356" Oct 30 02:00:38.033193 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:38.033171 6221 scope.go:122] "RemoveContainer" containerID="e6ef40e0cadc42602eb3ab9fee34501d925b04b58ff8d542e5a3d796150c1356" Oct 30 02:00:38.033514 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:00:38.033493 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6ef40e0cadc42602eb3ab9fee34501d925b04b58ff8d542e5a3d796150c1356\": container with ID starting with e6ef40e0cadc42602eb3ab9fee34501d925b04b58ff8d542e5a3d796150c1356 not found: ID does not exist" containerID="e6ef40e0cadc42602eb3ab9fee34501d925b04b58ff8d542e5a3d796150c1356" Oct 30 02:00:38.033600 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:38.033522 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6ef40e0cadc42602eb3ab9fee34501d925b04b58ff8d542e5a3d796150c1356"} err="failed to get container status \"e6ef40e0cadc42602eb3ab9fee34501d925b04b58ff8d542e5a3d796150c1356\": rpc error: code = NotFound desc = could not find container \"e6ef40e0cadc42602eb3ab9fee34501d925b04b58ff8d542e5a3d796150c1356\": container with ID starting with e6ef40e0cadc42602eb3ab9fee34501d925b04b58ff8d542e5a3d796150c1356 not found: ID does not exist" Oct 30 02:00:38.063064 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:38.063032 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm"] Oct 30 02:00:38.075329 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:38.075296 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-8488tmbnm"] Oct 30 02:00:38.101018 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:38.100982 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76466xfk4"] Oct 30 02:00:38.104186 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:38.104163 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-093d23/payload-processing-external-model-e2e-extmod-b-093d23-76466xfk4"] Oct 30 02:00:38.572557 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:38.571592 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07b7fc55-3b8b-4019-aae0-88231682d45e" path="/var/lib/kubelet/pods/07b7fc55-3b8b-4019-aae0-88231682d45e/volumes" Oct 30 02:00:38.572557 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:38.572358 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f51c2d68-f0ff-4721-be1b-076584606b7e" path="/var/lib/kubelet/pods/f51c2d68-f0ff-4721-be1b-076584606b7e/volumes" Oct 30 02:00:49.260226 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:00:49.260188 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 02:00:51.547389 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:00:51.547346 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 02:01:17.670154 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.670116 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbk7256"] Oct 30 02:01:17.670866 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.670850 6221 cpu_manager.go:414] "containerMap: removing container" podUID="76d7ecb2-375a-49ae-a766-db2b5405bfcd" containerName="cleanup" Oct 30 02:01:17.670921 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.670866 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76d7ecb2-375a-49ae-a766-db2b5405bfcd" containerName="cleanup" Oct 30 02:01:17.670921 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.670880 6221 cpu_manager.go:414] "containerMap: removing container" podUID="108b6f30-ce76-41e2-a888-1d5601c4eeea" containerName="cleanup" Oct 30 02:01:17.670921 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.670885 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="108b6f30-ce76-41e2-a888-1d5601c4eeea" containerName="cleanup" Oct 30 02:01:17.670921 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.670894 6221 cpu_manager.go:414] "containerMap: removing container" podUID="f51c2d68-f0ff-4721-be1b-076584606b7e" containerName="payload-processing" Oct 30 02:01:17.670921 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.670899 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f51c2d68-f0ff-4721-be1b-076584606b7e" containerName="payload-processing" Oct 30 02:01:17.670921 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.670916 6221 cpu_manager.go:414] "containerMap: removing container" podUID="07b7fc55-3b8b-4019-aae0-88231682d45e" containerName="payload-processing" Oct 30 02:01:17.670921 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.670921 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07b7fc55-3b8b-4019-aae0-88231682d45e" containerName="payload-processing" Oct 30 02:01:17.671145 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.670927 6221 cpu_manager.go:414] "containerMap: removing container" podUID="a451aa96-5403-409f-8a18-c1b2ea5a2231" containerName="cleanup" Oct 30 02:01:17.671145 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.670932 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a451aa96-5403-409f-8a18-c1b2ea5a2231" containerName="cleanup" Oct 30 02:01:17.671145 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.670938 6221 cpu_manager.go:414] "containerMap: removing container" podUID="381d1132-541d-4397-91ab-285a4b873991" containerName="cleanup" Oct 30 02:01:17.671145 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.670944 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="381d1132-541d-4397-91ab-285a4b873991" containerName="cleanup" Oct 30 02:01:17.671145 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.670955 6221 cpu_manager.go:414] "containerMap: removing container" podUID="f1038f98-dfbd-4839-b07f-2cb9742f144c" containerName="cleanup" Oct 30 02:01:17.671145 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.670959 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1038f98-dfbd-4839-b07f-2cb9742f144c" containerName="cleanup" Oct 30 02:01:17.671145 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.670968 6221 cpu_manager.go:414] "containerMap: removing container" podUID="a451aa96-5403-409f-8a18-c1b2ea5a2231" containerName="cleanup" Oct 30 02:01:17.671145 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.670972 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a451aa96-5403-409f-8a18-c1b2ea5a2231" containerName="cleanup" Oct 30 02:01:17.671145 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.670980 6221 cpu_manager.go:414] "containerMap: removing container" podUID="f1038f98-dfbd-4839-b07f-2cb9742f144c" containerName="cleanup" Oct 30 02:01:17.671145 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.670984 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1038f98-dfbd-4839-b07f-2cb9742f144c" containerName="cleanup" Oct 30 02:01:17.671145 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.670990 6221 cpu_manager.go:414] "containerMap: removing container" podUID="f1038f98-dfbd-4839-b07f-2cb9742f144c" containerName="cleanup" Oct 30 02:01:17.671145 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.670994 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1038f98-dfbd-4839-b07f-2cb9742f144c" containerName="cleanup" Oct 30 02:01:17.671145 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.671003 6221 cpu_manager.go:414] "containerMap: removing container" podUID="381d1132-541d-4397-91ab-285a4b873991" containerName="cleanup" Oct 30 02:01:17.671145 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.671037 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="381d1132-541d-4397-91ab-285a4b873991" containerName="cleanup" Oct 30 02:01:17.671145 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.671048 6221 cpu_manager.go:414] "containerMap: removing container" podUID="381d1132-541d-4397-91ab-285a4b873991" containerName="cleanup" Oct 30 02:01:17.671145 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.671052 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="381d1132-541d-4397-91ab-285a4b873991" containerName="cleanup" Oct 30 02:01:17.671145 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.671065 6221 cpu_manager.go:414] "containerMap: removing container" podUID="108b6f30-ce76-41e2-a888-1d5601c4eeea" containerName="cleanup" Oct 30 02:01:17.671145 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.671069 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="108b6f30-ce76-41e2-a888-1d5601c4eeea" containerName="cleanup" Oct 30 02:01:17.671145 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.671083 6221 cpu_manager.go:414] "containerMap: removing container" podUID="76d7ecb2-375a-49ae-a766-db2b5405bfcd" containerName="cleanup" Oct 30 02:01:17.671145 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.671087 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76d7ecb2-375a-49ae-a766-db2b5405bfcd" containerName="cleanup" Oct 30 02:01:17.671145 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.671095 6221 cpu_manager.go:414] "containerMap: removing container" podUID="a451aa96-5403-409f-8a18-c1b2ea5a2231" containerName="cleanup" Oct 30 02:01:17.671145 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.671100 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a451aa96-5403-409f-8a18-c1b2ea5a2231" containerName="cleanup" Oct 30 02:01:17.671145 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.671107 6221 cpu_manager.go:414] "containerMap: removing container" podUID="76d7ecb2-375a-49ae-a766-db2b5405bfcd" containerName="cleanup" Oct 30 02:01:17.671145 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.671111 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76d7ecb2-375a-49ae-a766-db2b5405bfcd" containerName="cleanup" Oct 30 02:01:17.671886 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.671217 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1038f98-dfbd-4839-b07f-2cb9742f144c" containerName="cleanup" Oct 30 02:01:17.671886 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.671225 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="381d1132-541d-4397-91ab-285a4b873991" containerName="cleanup" Oct 30 02:01:17.671886 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.671232 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="381d1132-541d-4397-91ab-285a4b873991" containerName="cleanup" Oct 30 02:01:17.671886 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.671239 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="108b6f30-ce76-41e2-a888-1d5601c4eeea" containerName="cleanup" Oct 30 02:01:17.671886 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.671247 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="76d7ecb2-375a-49ae-a766-db2b5405bfcd" containerName="cleanup" Oct 30 02:01:17.671886 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.671253 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="a451aa96-5403-409f-8a18-c1b2ea5a2231" containerName="cleanup" Oct 30 02:01:17.671886 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.671258 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="76d7ecb2-375a-49ae-a766-db2b5405bfcd" containerName="cleanup" Oct 30 02:01:17.671886 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.671267 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="76d7ecb2-375a-49ae-a766-db2b5405bfcd" containerName="cleanup" Oct 30 02:01:17.671886 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.671273 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="108b6f30-ce76-41e2-a888-1d5601c4eeea" containerName="cleanup" Oct 30 02:01:17.671886 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.671279 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="f51c2d68-f0ff-4721-be1b-076584606b7e" containerName="payload-processing" Oct 30 02:01:17.671886 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.671286 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="07b7fc55-3b8b-4019-aae0-88231682d45e" containerName="payload-processing" Oct 30 02:01:17.671886 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.671294 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1038f98-dfbd-4839-b07f-2cb9742f144c" containerName="cleanup" Oct 30 02:01:17.671886 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.671300 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="a451aa96-5403-409f-8a18-c1b2ea5a2231" containerName="cleanup" Oct 30 02:01:17.676686 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.676667 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbk7256" Oct 30 02:01:17.694762 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.694732 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbk7256"] Oct 30 02:01:17.701434 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.701408 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-92f0cc\"/\"agc-cred-2bf58c377925\"" Oct 30 02:01:17.701694 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.701427 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-92f0cc\"/\"kube-root-ca.crt\"" Oct 30 02:01:17.701824 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.701475 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-92f0cc\"/\"payload-processing-external-model-e2e--c7e40432-dockercfg-fffn4\"" Oct 30 02:01:17.701824 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.701589 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-92f0cc\"/\"openshift-service-ca.crt\"" Oct 30 02:01:17.704333 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.704309 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-92f0cc\"/\"routing-overlay\"" Oct 30 02:01:17.704460 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.704363 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-92f0cc\"/\"payload-processing-external-model-plugins-e2e-extmod-a-92f0cc\"" Oct 30 02:01:17.826501 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.826470 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spsxh\" (UniqueName: \"kubernetes.io/projected/97b696a3-338d-46ed-8490-3b6ae0bcd5d9-kube-api-access-spsxh\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-64fbk7256\" (UID: \"97b696a3-338d-46ed-8490-3b6ae0bcd5d9\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbk7256" Oct 30 02:01:17.826702 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.826621 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/97b696a3-338d-46ed-8490-3b6ae0bcd5d9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-64fbk7256\" (UID: \"97b696a3-338d-46ed-8490-3b6ae0bcd5d9\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbk7256" Oct 30 02:01:17.826862 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.826847 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/97b696a3-338d-46ed-8490-3b6ae0bcd5d9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-64fbk7256\" (UID: \"97b696a3-338d-46ed-8490-3b6ae0bcd5d9\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbk7256" Oct 30 02:01:17.826901 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.826872 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b696a3-338d-46ed-8490-3b6ae0bcd5d9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-64fbk7256\" (UID: \"97b696a3-338d-46ed-8490-3b6ae0bcd5d9\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbk7256" Oct 30 02:01:17.928714 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.928615 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/97b696a3-338d-46ed-8490-3b6ae0bcd5d9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-64fbk7256\" (UID: \"97b696a3-338d-46ed-8490-3b6ae0bcd5d9\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbk7256" Oct 30 02:01:17.928900 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.928867 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/97b696a3-338d-46ed-8490-3b6ae0bcd5d9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-64fbk7256\" (UID: \"97b696a3-338d-46ed-8490-3b6ae0bcd5d9\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbk7256" Oct 30 02:01:17.928988 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.928902 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b696a3-338d-46ed-8490-3b6ae0bcd5d9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-64fbk7256\" (UID: \"97b696a3-338d-46ed-8490-3b6ae0bcd5d9\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbk7256" Oct 30 02:01:17.928988 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.928960 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spsxh\" (UniqueName: \"kubernetes.io/projected/97b696a3-338d-46ed-8490-3b6ae0bcd5d9-kube-api-access-spsxh\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-64fbk7256\" (UID: \"97b696a3-338d-46ed-8490-3b6ae0bcd5d9\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbk7256" Oct 30 02:01:17.929424 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.929394 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/97b696a3-338d-46ed-8490-3b6ae0bcd5d9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-64fbk7256\" (UID: \"97b696a3-338d-46ed-8490-3b6ae0bcd5d9\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbk7256" Oct 30 02:01:17.929726 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.929707 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b696a3-338d-46ed-8490-3b6ae0bcd5d9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-64fbk7256\" (UID: \"97b696a3-338d-46ed-8490-3b6ae0bcd5d9\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbk7256" Oct 30 02:01:17.931338 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.931320 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/97b696a3-338d-46ed-8490-3b6ae0bcd5d9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-64fbk7256\" (UID: \"97b696a3-338d-46ed-8490-3b6ae0bcd5d9\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbk7256" Oct 30 02:01:17.949683 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.949651 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spsxh\" (UniqueName: \"kubernetes.io/projected/97b696a3-338d-46ed-8490-3b6ae0bcd5d9-kube-api-access-spsxh\") pod \"payload-processing-external-model-e2e-extmod-a-92f0cc-64fbk7256\" (UID: \"97b696a3-338d-46ed-8490-3b6ae0bcd5d9\") " pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbk7256" Oct 30 02:01:17.991715 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:17.991674 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbk7256" Oct 30 02:01:18.303609 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:18.303575 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbk7256"] Oct 30 02:01:18.305826 ip-10-0-135-65 kubenswrapper[6221]: W1030 02:01:18.305796 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97b696a3_338d_46ed_8490_3b6ae0bcd5d9.slice/crio-257c04ffc294c212749161a1844589f87eef0dc14d8d2c890f55ec8f1a60c84f WatchSource:0}: Error finding container 257c04ffc294c212749161a1844589f87eef0dc14d8d2c890f55ec8f1a60c84f: Status 404 returned error can't find the container with id 257c04ffc294c212749161a1844589f87eef0dc14d8d2c890f55ec8f1a60c84f Oct 30 02:01:18.447224 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:18.447175 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbk7256" event={"ID":"97b696a3-338d-46ed-8490-3b6ae0bcd5d9","Type":"ContainerStarted","Data":"216f7a336e3da94c6801b0f83e5f36b6a7e8860883d8269c4a65e0747ec91105"} Oct 30 02:01:18.447438 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:18.447239 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbk7256" event={"ID":"97b696a3-338d-46ed-8490-3b6ae0bcd5d9","Type":"ContainerStarted","Data":"257c04ffc294c212749161a1844589f87eef0dc14d8d2c890f55ec8f1a60c84f"} Oct 30 02:01:18.447438 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:18.447309 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbk7256" Oct 30 02:01:18.489173 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:18.489122 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbk7256" podStartSLOduration=1.489107328 podStartE2EDuration="1.489107328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:01:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:01:18.487097865 +0000 UTC m=+2602.065385211" watchObservedRunningTime="2026-10-30 02:01:18.489107328 +0000 UTC m=+2602.067394665" Oct 30 02:01:21.498016 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:01:21.497976 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 02:01:29.459222 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:29.459175 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbk7256" Oct 30 02:01:29.717365 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:29.717273 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbk7256"] Oct 30 02:01:34.717797 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:34.717724 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbk7256" podUID="97b696a3-338d-46ed-8490-3b6ae0bcd5d9" containerName="payload-processing" containerID="cri-o://216f7a336e3da94c6801b0f83e5f36b6a7e8860883d8269c4a65e0747ec91105" gracePeriod=25 Oct 30 02:01:35.014970 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:35.014938 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbk7256" Oct 30 02:01:35.124201 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:35.124161 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97b696a3-338d-46ed-8490-3b6ae0bcd5d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b696a3-338d-46ed-8490-3b6ae0bcd5d9-plugins-config-volume\") pod \"97b696a3-338d-46ed-8490-3b6ae0bcd5d9\" (UID: \"97b696a3-338d-46ed-8490-3b6ae0bcd5d9\") " Oct 30 02:01:35.124385 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:35.124230 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97b696a3-338d-46ed-8490-3b6ae0bcd5d9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/97b696a3-338d-46ed-8490-3b6ae0bcd5d9-routing-overlay-volume\") pod \"97b696a3-338d-46ed-8490-3b6ae0bcd5d9\" (UID: \"97b696a3-338d-46ed-8490-3b6ae0bcd5d9\") " Oct 30 02:01:35.124385 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:35.124272 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97b696a3-338d-46ed-8490-3b6ae0bcd5d9-kube-api-access-spsxh\" (UniqueName: \"kubernetes.io/projected/97b696a3-338d-46ed-8490-3b6ae0bcd5d9-kube-api-access-spsxh\") pod \"97b696a3-338d-46ed-8490-3b6ae0bcd5d9\" (UID: \"97b696a3-338d-46ed-8490-3b6ae0bcd5d9\") " Oct 30 02:01:35.124385 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:35.124376 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97b696a3-338d-46ed-8490-3b6ae0bcd5d9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/97b696a3-338d-46ed-8490-3b6ae0bcd5d9-provider-credentials\") pod \"97b696a3-338d-46ed-8490-3b6ae0bcd5d9\" (UID: \"97b696a3-338d-46ed-8490-3b6ae0bcd5d9\") " Oct 30 02:01:35.124689 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:35.124663 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97b696a3-338d-46ed-8490-3b6ae0bcd5d9-routing-overlay-volume" pod "97b696a3-338d-46ed-8490-3b6ae0bcd5d9" (UID: "97b696a3-338d-46ed-8490-3b6ae0bcd5d9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:01:35.124755 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:35.124691 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97b696a3-338d-46ed-8490-3b6ae0bcd5d9-plugins-config-volume" pod "97b696a3-338d-46ed-8490-3b6ae0bcd5d9" (UID: "97b696a3-338d-46ed-8490-3b6ae0bcd5d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:01:35.125012 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:35.124995 6221 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/97b696a3-338d-46ed-8490-3b6ae0bcd5d9-routing-overlay-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:01:35.125055 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:35.125015 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b696a3-338d-46ed-8490-3b6ae0bcd5d9-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:01:35.126522 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:35.126500 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97b696a3-338d-46ed-8490-3b6ae0bcd5d9-provider-credentials" pod "97b696a3-338d-46ed-8490-3b6ae0bcd5d9" (UID: "97b696a3-338d-46ed-8490-3b6ae0bcd5d9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:35.126602 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:35.126518 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97b696a3-338d-46ed-8490-3b6ae0bcd5d9-kube-api-access-spsxh" pod "97b696a3-338d-46ed-8490-3b6ae0bcd5d9" (UID: "97b696a3-338d-46ed-8490-3b6ae0bcd5d9"). InnerVolumeSpecName "kube-api-access-spsxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:01:35.227026 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:35.226980 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spsxh\" (UniqueName: \"kubernetes.io/projected/97b696a3-338d-46ed-8490-3b6ae0bcd5d9-kube-api-access-spsxh\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:01:35.227026 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:35.227016 6221 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/97b696a3-338d-46ed-8490-3b6ae0bcd5d9-provider-credentials\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:01:35.586703 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:35.586667 6221 generic.go:356] "Generic (PLEG): container finished" podID="97b696a3-338d-46ed-8490-3b6ae0bcd5d9" containerID="216f7a336e3da94c6801b0f83e5f36b6a7e8860883d8269c4a65e0747ec91105" exitCode=0 Oct 30 02:01:35.586922 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:35.586712 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbk7256" Oct 30 02:01:35.586922 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:35.586755 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbk7256" event={"ID":"97b696a3-338d-46ed-8490-3b6ae0bcd5d9","Type":"ContainerDied","Data":"216f7a336e3da94c6801b0f83e5f36b6a7e8860883d8269c4a65e0747ec91105"} Oct 30 02:01:35.586922 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:35.586819 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbk7256" event={"ID":"97b696a3-338d-46ed-8490-3b6ae0bcd5d9","Type":"ContainerDied","Data":"257c04ffc294c212749161a1844589f87eef0dc14d8d2c890f55ec8f1a60c84f"} Oct 30 02:01:35.586922 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:35.586848 6221 scope.go:122] "RemoveContainer" containerID="216f7a336e3da94c6801b0f83e5f36b6a7e8860883d8269c4a65e0747ec91105" Oct 30 02:01:35.602479 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:35.602458 6221 scope.go:122] "RemoveContainer" containerID="216f7a336e3da94c6801b0f83e5f36b6a7e8860883d8269c4a65e0747ec91105" Oct 30 02:01:35.602850 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:01:35.602826 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"216f7a336e3da94c6801b0f83e5f36b6a7e8860883d8269c4a65e0747ec91105\": container with ID starting with 216f7a336e3da94c6801b0f83e5f36b6a7e8860883d8269c4a65e0747ec91105 not found: ID does not exist" containerID="216f7a336e3da94c6801b0f83e5f36b6a7e8860883d8269c4a65e0747ec91105" Oct 30 02:01:35.603010 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:35.602860 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"216f7a336e3da94c6801b0f83e5f36b6a7e8860883d8269c4a65e0747ec91105"} err="failed to get container status \"216f7a336e3da94c6801b0f83e5f36b6a7e8860883d8269c4a65e0747ec91105\": rpc error: code = NotFound desc = could not find container \"216f7a336e3da94c6801b0f83e5f36b6a7e8860883d8269c4a65e0747ec91105\": container with ID starting with 216f7a336e3da94c6801b0f83e5f36b6a7e8860883d8269c4a65e0747ec91105 not found: ID does not exist" Oct 30 02:01:35.639894 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:35.639860 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbk7256"] Oct 30 02:01:35.652142 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:35.652107 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-92f0cc/payload-processing-external-model-e2e-extmod-a-92f0cc-64fbk7256"] Oct 30 02:01:36.555166 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:36.555131 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97b696a3-338d-46ed-8490-3b6ae0bcd5d9" path="/var/lib/kubelet/pods/97b696a3-338d-46ed-8490-3b6ae0bcd5d9/volumes" Oct 30 02:01:45.087638 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:01:45.087544 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 02:01:51.496965 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:01:51.496924 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 02:02:21.505785 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:02:21.505747 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 02:02:36.536511 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:02:36.536475 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 02:02:51.496451 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:02:51.496411 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 02:02:57.072326 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:02:57.072298 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2vsns_d332bbb4-c699-44e8-96ba-516d8ca03e95/ovn-acl-logging/0.log" Oct 30 02:02:57.075820 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:02:57.075797 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2vsns_d332bbb4-c699-44e8-96ba-516d8ca03e95/ovn-acl-logging/0.log" Oct 30 02:03:15.209413 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:03:15.209331 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 02:03:21.495674 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:03:21.495636 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 02:03:51.496307 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:03:51.496268 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 02:04:14.791974 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:04:14.791935 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 02:04:21.495844 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:04:21.495803 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 02:04:40.958342 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:04:40.958254 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-717065-69d69c879f-lxc5m"] Oct 30 02:04:40.959085 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:04:40.959067 6221 cpu_manager.go:414] "containerMap: removing container" podUID="108b6f30-ce76-41e2-a888-1d5601c4eeea" containerName="cleanup" Oct 30 02:04:40.959140 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:04:40.959087 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="108b6f30-ce76-41e2-a888-1d5601c4eeea" containerName="cleanup" Oct 30 02:04:40.959140 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:04:40.959132 6221 cpu_manager.go:414] "containerMap: removing container" podUID="97b696a3-338d-46ed-8490-3b6ae0bcd5d9" containerName="payload-processing" Oct 30 02:04:40.959140 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:04:40.959137 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97b696a3-338d-46ed-8490-3b6ae0bcd5d9" containerName="payload-processing" Oct 30 02:04:40.959245 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:04:40.959240 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="97b696a3-338d-46ed-8490-3b6ae0bcd5d9" containerName="payload-processing" Oct 30 02:04:40.959279 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:04:40.959255 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="108b6f30-ce76-41e2-a888-1d5601c4eeea" containerName="cleanup" Oct 30 02:04:40.959279 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:04:40.959265 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="a451aa96-5403-409f-8a18-c1b2ea5a2231" containerName="cleanup" Oct 30 02:04:40.959279 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:04:40.959272 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1038f98-dfbd-4839-b07f-2cb9742f144c" containerName="cleanup" Oct 30 02:04:40.959373 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:04:40.959280 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="381d1132-541d-4397-91ab-285a4b873991" containerName="cleanup" Oct 30 02:04:40.962659 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:04:40.962637 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-717065-69d69c879f-lxc5m" Oct 30 02:04:40.981348 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:04:40.981318 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-717065-69d69c879f-lxc5m"] Oct 30 02:04:40.989949 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:04:40.989893 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-717065-dockercfg-g9sg9\"" Oct 30 02:04:40.989949 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:04:40.989914 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-717065\"" Oct 30 02:04:41.120552 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:04:41.119940 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x62mw\" (UniqueName: \"kubernetes.io/projected/5b74befa-31a1-4198-80a6-a46db3294eaa-kube-api-access-x62mw\") pod \"payload-pre-processing-e2e-inf-a-717065-69d69c879f-lxc5m\" (UID: \"5b74befa-31a1-4198-80a6-a46db3294eaa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-717065-69d69c879f-lxc5m" Oct 30 02:04:41.120552 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:04:41.120097 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b74befa-31a1-4198-80a6-a46db3294eaa-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-717065-69d69c879f-lxc5m\" (UID: \"5b74befa-31a1-4198-80a6-a46db3294eaa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-717065-69d69c879f-lxc5m" Oct 30 02:04:41.236608 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:04:41.223467 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x62mw\" (UniqueName: \"kubernetes.io/projected/5b74befa-31a1-4198-80a6-a46db3294eaa-kube-api-access-x62mw\") pod \"payload-pre-processing-e2e-inf-a-717065-69d69c879f-lxc5m\" (UID: \"5b74befa-31a1-4198-80a6-a46db3294eaa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-717065-69d69c879f-lxc5m" Oct 30 02:04:41.248870 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:04:41.245836 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b74befa-31a1-4198-80a6-a46db3294eaa-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-717065-69d69c879f-lxc5m\" (UID: \"5b74befa-31a1-4198-80a6-a46db3294eaa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-717065-69d69c879f-lxc5m" Oct 30 02:04:41.249467 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:04:41.248573 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b74befa-31a1-4198-80a6-a46db3294eaa-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-717065-69d69c879f-lxc5m\" (UID: \"5b74befa-31a1-4198-80a6-a46db3294eaa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-717065-69d69c879f-lxc5m" Oct 30 02:04:41.265891 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:04:41.265855 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x62mw\" (UniqueName: \"kubernetes.io/projected/5b74befa-31a1-4198-80a6-a46db3294eaa-kube-api-access-x62mw\") pod \"payload-pre-processing-e2e-inf-a-717065-69d69c879f-lxc5m\" (UID: \"5b74befa-31a1-4198-80a6-a46db3294eaa\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-717065-69d69c879f-lxc5m" Oct 30 02:04:41.277589 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:04:41.276729 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-717065-69d69c879f-lxc5m" Oct 30 02:04:41.592596 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:04:41.592563 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-717065-69d69c879f-lxc5m"] Oct 30 02:04:41.594836 ip-10-0-135-65 kubenswrapper[6221]: W1030 02:04:41.594799 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b74befa_31a1_4198_80a6_a46db3294eaa.slice/crio-54a9a4b9ccabb0d71156eb42cbd4ef56237976166f9e82403d8215e6796a976c WatchSource:0}: Error finding container 54a9a4b9ccabb0d71156eb42cbd4ef56237976166f9e82403d8215e6796a976c: Status 404 returned error can't find the container with id 54a9a4b9ccabb0d71156eb42cbd4ef56237976166f9e82403d8215e6796a976c Oct 30 02:04:42.277147 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:04:42.277108 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-717065-69d69c879f-lxc5m" event={"ID":"5b74befa-31a1-4198-80a6-a46db3294eaa","Type":"ContainerStarted","Data":"fae51e8a708bd5488ca610e54f9e390cd9cbc40faa6ee850494e4e3492e4b6f7"} Oct 30 02:04:42.277630 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:04:42.277168 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-717065-69d69c879f-lxc5m" event={"ID":"5b74befa-31a1-4198-80a6-a46db3294eaa","Type":"ContainerStarted","Data":"54a9a4b9ccabb0d71156eb42cbd4ef56237976166f9e82403d8215e6796a976c"} Oct 30 02:04:42.277630 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:04:42.277213 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-717065-69d69c879f-lxc5m" Oct 30 02:04:42.322014 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:04:42.321961 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-717065-69d69c879f-lxc5m" podStartSLOduration=2.321945894 podStartE2EDuration="2.321945894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:04:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:04:42.319133287 +0000 UTC m=+2805.897420624" watchObservedRunningTime="2026-10-30 02:04:42.321945894 +0000 UTC m=+2805.900233231" Oct 30 02:04:51.495173 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:04:51.495131 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 02:04:53.288338 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:04:53.288303 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-717065-69d69c879f-lxc5m" Oct 30 02:04:58.107725 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:04:58.107692 6221 scope.go:122] "RemoveContainer" containerID="d1409f039eddf0c078e1b0be630966ed194f3cdbc88b9883b41951df654c83bd" Oct 30 02:05:05.464507 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:05:05.464469 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 02:05:20.776225 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:05:20.776190 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-717065-7d598477fd-zrzd7"] Oct 30 02:05:20.782144 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:05:20.782118 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-717065-7d598477fd-zrzd7" Oct 30 02:05:20.800265 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:05:20.800235 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-717065-7d598477fd-zrzd7"] Oct 30 02:05:20.808596 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:05:20.808571 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-717065-serving-cert\"" Oct 30 02:05:20.940698 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:05:20.940655 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4d33590-0112-4d25-b383-a7b6757d480d-metrics-certs\") pod \"maas-api-e2e-inf-a-717065-7d598477fd-zrzd7\" (UID: \"e4d33590-0112-4d25-b383-a7b6757d480d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-717065-7d598477fd-zrzd7" Oct 30 02:05:20.940884 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:05:20.940838 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4d33590-0112-4d25-b383-a7b6757d480d-maas-api-tls\") pod \"maas-api-e2e-inf-a-717065-7d598477fd-zrzd7\" (UID: \"e4d33590-0112-4d25-b383-a7b6757d480d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-717065-7d598477fd-zrzd7" Oct 30 02:05:20.941045 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:05:20.941026 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4d33590-0112-4d25-b383-a7b6757d480d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-717065-7d598477fd-zrzd7\" (UID: \"e4d33590-0112-4d25-b383-a7b6757d480d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-717065-7d598477fd-zrzd7" Oct 30 02:05:20.941127 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:05:20.941062 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zgch\" (UniqueName: \"kubernetes.io/projected/e4d33590-0112-4d25-b383-a7b6757d480d-kube-api-access-9zgch\") pod \"maas-api-e2e-inf-a-717065-7d598477fd-zrzd7\" (UID: \"e4d33590-0112-4d25-b383-a7b6757d480d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-717065-7d598477fd-zrzd7" Oct 30 02:05:21.043050 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:05:21.043009 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4d33590-0112-4d25-b383-a7b6757d480d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-717065-7d598477fd-zrzd7\" (UID: \"e4d33590-0112-4d25-b383-a7b6757d480d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-717065-7d598477fd-zrzd7" Oct 30 02:05:21.043266 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:05:21.043066 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zgch\" (UniqueName: \"kubernetes.io/projected/e4d33590-0112-4d25-b383-a7b6757d480d-kube-api-access-9zgch\") pod \"maas-api-e2e-inf-a-717065-7d598477fd-zrzd7\" (UID: \"e4d33590-0112-4d25-b383-a7b6757d480d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-717065-7d598477fd-zrzd7" Oct 30 02:05:21.043266 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:05:21.043136 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4d33590-0112-4d25-b383-a7b6757d480d-metrics-certs\") pod \"maas-api-e2e-inf-a-717065-7d598477fd-zrzd7\" (UID: \"e4d33590-0112-4d25-b383-a7b6757d480d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-717065-7d598477fd-zrzd7" Oct 30 02:05:21.043466 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:05:21.043390 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4d33590-0112-4d25-b383-a7b6757d480d-maas-api-tls\") pod \"maas-api-e2e-inf-a-717065-7d598477fd-zrzd7\" (UID: \"e4d33590-0112-4d25-b383-a7b6757d480d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-717065-7d598477fd-zrzd7" Oct 30 02:05:21.043610 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:05:21.043591 6221 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-717065-serving-cert: secret "maas-api-e2e-inf-a-717065-serving-cert" not found Oct 30 02:05:21.043695 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:05:21.043665 6221 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4d33590-0112-4d25-b383-a7b6757d480d-maas-api-tls podName:e4d33590-0112-4d25-b383-a7b6757d480d nodeName:}" failed. No retries permitted until 2026-10-30 02:05:21.543645823 +0000 UTC m=+2845.121933140 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e4d33590-0112-4d25-b383-a7b6757d480d-maas-api-tls") pod "maas-api-e2e-inf-a-717065-7d598477fd-zrzd7" (UID: "e4d33590-0112-4d25-b383-a7b6757d480d") : secret "maas-api-e2e-inf-a-717065-serving-cert" not found Oct 30 02:05:21.043906 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:05:21.043880 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4d33590-0112-4d25-b383-a7b6757d480d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-717065-7d598477fd-zrzd7\" (UID: \"e4d33590-0112-4d25-b383-a7b6757d480d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-717065-7d598477fd-zrzd7" Oct 30 02:05:21.046407 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:05:21.046371 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4d33590-0112-4d25-b383-a7b6757d480d-metrics-certs\") pod \"maas-api-e2e-inf-a-717065-7d598477fd-zrzd7\" (UID: \"e4d33590-0112-4d25-b383-a7b6757d480d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-717065-7d598477fd-zrzd7" Oct 30 02:05:21.063875 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:05:21.063846 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zgch\" (UniqueName: \"kubernetes.io/projected/e4d33590-0112-4d25-b383-a7b6757d480d-kube-api-access-9zgch\") pod \"maas-api-e2e-inf-a-717065-7d598477fd-zrzd7\" (UID: \"e4d33590-0112-4d25-b383-a7b6757d480d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-717065-7d598477fd-zrzd7" Oct 30 02:05:21.495996 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:05:21.495913 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 02:05:21.551955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:05:21.551924 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4d33590-0112-4d25-b383-a7b6757d480d-maas-api-tls\") pod \"maas-api-e2e-inf-a-717065-7d598477fd-zrzd7\" (UID: \"e4d33590-0112-4d25-b383-a7b6757d480d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-717065-7d598477fd-zrzd7" Oct 30 02:05:21.554675 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:05:21.554651 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4d33590-0112-4d25-b383-a7b6757d480d-maas-api-tls\") pod \"maas-api-e2e-inf-a-717065-7d598477fd-zrzd7\" (UID: \"e4d33590-0112-4d25-b383-a7b6757d480d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-717065-7d598477fd-zrzd7" Oct 30 02:05:21.696652 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:05:21.696617 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-717065-7d598477fd-zrzd7" Oct 30 02:05:22.019250 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:05:22.019169 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-717065-7d598477fd-zrzd7"] Oct 30 02:05:22.023677 ip-10-0-135-65 kubenswrapper[6221]: W1030 02:05:22.023651 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4d33590_0112_4d25_b383_a7b6757d480d.slice/crio-5e95bd553cf556b2b814ba6ceac39b228447b1ceb9f3a7f524d60566ed9636e8 WatchSource:0}: Error finding container 5e95bd553cf556b2b814ba6ceac39b228447b1ceb9f3a7f524d60566ed9636e8: Status 404 returned error can't find the container with id 5e95bd553cf556b2b814ba6ceac39b228447b1ceb9f3a7f524d60566ed9636e8 Oct 30 02:05:22.025335 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:05:22.025316 6221 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 02:05:22.643148 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:05:22.643112 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-717065-7d598477fd-zrzd7" event={"ID":"e4d33590-0112-4d25-b383-a7b6757d480d","Type":"ContainerStarted","Data":"28fd6d46905190e83831dc0fc83c1822908f2fd89da34e6c23662903a24c49a6"} Oct 30 02:05:22.643325 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:05:22.643166 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-717065-7d598477fd-zrzd7" event={"ID":"e4d33590-0112-4d25-b383-a7b6757d480d","Type":"ContainerStarted","Data":"5e95bd553cf556b2b814ba6ceac39b228447b1ceb9f3a7f524d60566ed9636e8"} Oct 30 02:05:22.643325 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:05:22.643213 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-717065-7d598477fd-zrzd7" Oct 30 02:05:22.690216 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:05:22.690165 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-717065-7d598477fd-zrzd7" podStartSLOduration=2.2586080060000002 podStartE2EDuration="2.690152374s" totalImagesPullingTime="431.544368ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:05:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:05:22.688014494 +0000 UTC m=+2846.266301831" watchObservedRunningTime="2026-10-30 02:05:22.690152374 +0000 UTC m=+2846.268439709" Oct 30 02:05:28.655995 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:05:28.655961 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-717065-7d598477fd-zrzd7" Oct 30 02:05:44.903075 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:05:44.891328 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 02:05:51.495913 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:05:51.495867 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 02:06:10.916720 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:10.916628 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f55r9vm"] Oct 30 02:06:10.921375 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:10.921352 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f55r9vm" Oct 30 02:06:10.938269 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:10.938233 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-76a19e\"/\"agc-cred-139046a94a22\"" Oct 30 02:06:10.942488 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:10.942452 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f55r9vm"] Oct 30 02:06:10.950008 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:10.949970 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-76a19e\"/\"kube-root-ca.crt\"" Oct 30 02:06:10.950008 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:10.949970 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-76a19e\"/\"payload-processing-external-model-plugins-e2e-extmod-a-76a19e\"" Oct 30 02:06:10.950275 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:10.950033 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-76a19e\"/\"openshift-service-ca.crt\"" Oct 30 02:06:10.950275 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:10.950079 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-76a19e\"/\"routing-overlay\"" Oct 30 02:06:10.950811 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:10.950782 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-76a19e\"/\"payload-processing-external-model-e2e--70e20018-dockercfg-cvwmc\"" Oct 30 02:06:11.062174 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:11.062133 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5ce2fe8-4a3f-4a02-a357-5e24226b12ec-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-77f55r9vm\" (UID: \"d5ce2fe8-4a3f-4a02-a357-5e24226b12ec\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f55r9vm" Oct 30 02:06:11.062399 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:11.062191 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d5ce2fe8-4a3f-4a02-a357-5e24226b12ec-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-77f55r9vm\" (UID: \"d5ce2fe8-4a3f-4a02-a357-5e24226b12ec\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f55r9vm" Oct 30 02:06:11.062399 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:11.062252 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlrhd\" (UniqueName: \"kubernetes.io/projected/d5ce2fe8-4a3f-4a02-a357-5e24226b12ec-kube-api-access-jlrhd\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-77f55r9vm\" (UID: \"d5ce2fe8-4a3f-4a02-a357-5e24226b12ec\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f55r9vm" Oct 30 02:06:11.062399 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:11.062273 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d5ce2fe8-4a3f-4a02-a357-5e24226b12ec-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-77f55r9vm\" (UID: \"d5ce2fe8-4a3f-4a02-a357-5e24226b12ec\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f55r9vm" Oct 30 02:06:11.163605 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:11.163563 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5ce2fe8-4a3f-4a02-a357-5e24226b12ec-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-77f55r9vm\" (UID: \"d5ce2fe8-4a3f-4a02-a357-5e24226b12ec\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f55r9vm" Oct 30 02:06:11.163837 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:11.163617 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d5ce2fe8-4a3f-4a02-a357-5e24226b12ec-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-77f55r9vm\" (UID: \"d5ce2fe8-4a3f-4a02-a357-5e24226b12ec\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f55r9vm" Oct 30 02:06:11.163837 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:11.163675 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlrhd\" (UniqueName: \"kubernetes.io/projected/d5ce2fe8-4a3f-4a02-a357-5e24226b12ec-kube-api-access-jlrhd\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-77f55r9vm\" (UID: \"d5ce2fe8-4a3f-4a02-a357-5e24226b12ec\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f55r9vm" Oct 30 02:06:11.163837 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:11.163694 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d5ce2fe8-4a3f-4a02-a357-5e24226b12ec-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-77f55r9vm\" (UID: \"d5ce2fe8-4a3f-4a02-a357-5e24226b12ec\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f55r9vm" Oct 30 02:06:11.164409 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:11.164384 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d5ce2fe8-4a3f-4a02-a357-5e24226b12ec-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-77f55r9vm\" (UID: \"d5ce2fe8-4a3f-4a02-a357-5e24226b12ec\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f55r9vm" Oct 30 02:06:11.164542 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:11.164421 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5ce2fe8-4a3f-4a02-a357-5e24226b12ec-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-77f55r9vm\" (UID: \"d5ce2fe8-4a3f-4a02-a357-5e24226b12ec\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f55r9vm" Oct 30 02:06:11.166239 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:11.166217 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d5ce2fe8-4a3f-4a02-a357-5e24226b12ec-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-77f55r9vm\" (UID: \"d5ce2fe8-4a3f-4a02-a357-5e24226b12ec\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f55r9vm" Oct 30 02:06:11.185124 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:11.185035 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlrhd\" (UniqueName: \"kubernetes.io/projected/d5ce2fe8-4a3f-4a02-a357-5e24226b12ec-kube-api-access-jlrhd\") pod \"payload-processing-external-model-e2e-extmod-a-76a19e-77f55r9vm\" (UID: \"d5ce2fe8-4a3f-4a02-a357-5e24226b12ec\") " pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f55r9vm" Oct 30 02:06:11.243912 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:11.243853 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f55r9vm" Oct 30 02:06:11.813896 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:11.813856 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f55r9vm"] Oct 30 02:06:12.073820 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:12.073725 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f55r9vm" event={"ID":"d5ce2fe8-4a3f-4a02-a357-5e24226b12ec","Type":"ContainerStarted","Data":"7b803ca3c7db95470e9a55e3eea740093a2ac5ee286d1862a63c49ae210ca4dd"} Oct 30 02:06:12.073820 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:12.073781 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f55r9vm" event={"ID":"d5ce2fe8-4a3f-4a02-a357-5e24226b12ec","Type":"ContainerStarted","Data":"9f9f341a7a2a66cfd4efe11a87758dc9a229d8da8fe905c113eec44f977f2c0f"} Oct 30 02:06:12.073820 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:12.073808 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f55r9vm" Oct 30 02:06:12.135305 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:12.135237 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f55r9vm" podStartSLOduration=2.135221931 podStartE2EDuration="2.135221931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:06:12.132358601 +0000 UTC m=+2895.710645934" watchObservedRunningTime="2026-10-30 02:06:12.135221931 +0000 UTC m=+2895.713509266" Oct 30 02:06:14.166554 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:14.157787 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f81908-55df8db8bb-dlxz9"] Oct 30 02:06:14.194644 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:14.172776 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f81908-55df8db8bb-dlxz9" Oct 30 02:06:14.201066 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:14.200986 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f81908-55df8db8bb-dlxz9"] Oct 30 02:06:14.202888 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:14.202690 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-f81908\"" Oct 30 02:06:14.202888 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:14.202844 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-f81908-dockercfg-brsc5\"" Oct 30 02:06:14.414254 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:14.414204 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfpqb\" (UniqueName: \"kubernetes.io/projected/3af28e5b-f8a6-46b4-af05-f8ea92975c01-kube-api-access-lfpqb\") pod \"payload-processing-e2e-inf-b-f81908-55df8db8bb-dlxz9\" (UID: \"3af28e5b-f8a6-46b4-af05-f8ea92975c01\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f81908-55df8db8bb-dlxz9" Oct 30 02:06:14.414441 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:14.414349 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3af28e5b-f8a6-46b4-af05-f8ea92975c01-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f81908-55df8db8bb-dlxz9\" (UID: \"3af28e5b-f8a6-46b4-af05-f8ea92975c01\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f81908-55df8db8bb-dlxz9" Oct 30 02:06:14.516612 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:14.516484 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfpqb\" (UniqueName: \"kubernetes.io/projected/3af28e5b-f8a6-46b4-af05-f8ea92975c01-kube-api-access-lfpqb\") pod \"payload-processing-e2e-inf-b-f81908-55df8db8bb-dlxz9\" (UID: \"3af28e5b-f8a6-46b4-af05-f8ea92975c01\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f81908-55df8db8bb-dlxz9" Oct 30 02:06:14.516813 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:14.516609 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3af28e5b-f8a6-46b4-af05-f8ea92975c01-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f81908-55df8db8bb-dlxz9\" (UID: \"3af28e5b-f8a6-46b4-af05-f8ea92975c01\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f81908-55df8db8bb-dlxz9" Oct 30 02:06:14.517350 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:14.517328 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3af28e5b-f8a6-46b4-af05-f8ea92975c01-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f81908-55df8db8bb-dlxz9\" (UID: \"3af28e5b-f8a6-46b4-af05-f8ea92975c01\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f81908-55df8db8bb-dlxz9" Oct 30 02:06:14.538187 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:14.538156 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfpqb\" (UniqueName: \"kubernetes.io/projected/3af28e5b-f8a6-46b4-af05-f8ea92975c01-kube-api-access-lfpqb\") pod \"payload-processing-e2e-inf-b-f81908-55df8db8bb-dlxz9\" (UID: \"3af28e5b-f8a6-46b4-af05-f8ea92975c01\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f81908-55df8db8bb-dlxz9" Oct 30 02:06:14.805139 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:14.805103 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f81908-55df8db8bb-dlxz9" Oct 30 02:06:15.160302 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:15.160272 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f81908-55df8db8bb-dlxz9"] Oct 30 02:06:15.161796 ip-10-0-135-65 kubenswrapper[6221]: W1030 02:06:15.161766 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3af28e5b_f8a6_46b4_af05_f8ea92975c01.slice/crio-e16676b3b6f2a6b4360b226b9b523604c241b5b68da03a544151f575a5b59b94 WatchSource:0}: Error finding container e16676b3b6f2a6b4360b226b9b523604c241b5b68da03a544151f575a5b59b94: Status 404 returned error can't find the container with id e16676b3b6f2a6b4360b226b9b523604c241b5b68da03a544151f575a5b59b94 Oct 30 02:06:15.177163 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:15.177119 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f81908-55df8db8bb-dlxz9" event={"ID":"3af28e5b-f8a6-46b4-af05-f8ea92975c01","Type":"ContainerStarted","Data":"e16676b3b6f2a6b4360b226b9b523604c241b5b68da03a544151f575a5b59b94"} Oct 30 02:06:16.186406 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:16.186368 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f81908-55df8db8bb-dlxz9" event={"ID":"3af28e5b-f8a6-46b4-af05-f8ea92975c01","Type":"ContainerStarted","Data":"824807c5eda587ce9b3b7a2c51fd761c2abacdce8cdc1230f74f98ce1aee6296"} Oct 30 02:06:16.186877 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:16.186449 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f81908-55df8db8bb-dlxz9" Oct 30 02:06:16.230987 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:16.230935 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-f81908-55df8db8bb-dlxz9" podStartSLOduration=2.230917989 podStartE2EDuration="2.230917989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:06:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:06:16.229272015 +0000 UTC m=+2899.807559351" watchObservedRunningTime="2026-10-30 02:06:16.230917989 +0000 UTC m=+2899.809205324" Oct 30 02:06:21.497117 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:06:21.497079 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 02:06:23.098810 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:23.098781 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f55r9vm" Oct 30 02:06:24.262613 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:24.262581 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 02:06:27.197465 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:27.197435 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f81908-55df8db8bb-dlxz9" Oct 30 02:06:34.166667 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:34.166625 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f55r9vm"] Oct 30 02:06:37.907639 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:37.907602 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5"] Oct 30 02:06:37.908098 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:37.907848 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5" podUID="23246593-d27c-4d51-a771-ae5b61df6fbb" containerName="istio-proxy" containerID="cri-o://341ea0673c6deb48b8411452f7544c7454366302f9c16bfa3c92d9cccb0fe71d" gracePeriod=30 Oct 30 02:06:39.176346 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:39.176268 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f55r9vm" podUID="d5ce2fe8-4a3f-4a02-a357-5e24226b12ec" containerName="payload-processing" containerID="cri-o://7b803ca3c7db95470e9a55e3eea740093a2ac5ee286d1862a63c49ae210ca4dd" gracePeriod=25 Oct 30 02:06:39.193389 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:39.193174 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9"] Oct 30 02:06:39.194283 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:39.194246 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" podUID="48557095-c2dc-40cb-8ee3-8ee71567c61c" containerName="istio-proxy" containerID="cri-o://02b28bf6299d286f6ec9a31d31008878b76c5e36c7bafa1122a21fa124e1219f" gracePeriod=30 Oct 30 02:06:39.414278 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:39.413472 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d"] Oct 30 02:06:39.414278 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:39.413771 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" podUID="56476948-dc26-41d2-9df4-e23f8a293d75" containerName="istio-proxy" containerID="cri-o://77b25c8d03f17e9acf7d3fbe75a82ebf5d7a1a630fbd5420d9e588dcea5a5f01" gracePeriod=30 Oct 30 02:06:39.505309 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:39.505276 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f55r9vm" Oct 30 02:06:39.601208 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:39.601170 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5ce2fe8-4a3f-4a02-a357-5e24226b12ec-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d5ce2fe8-4a3f-4a02-a357-5e24226b12ec-routing-overlay-volume\") pod \"d5ce2fe8-4a3f-4a02-a357-5e24226b12ec\" (UID: \"d5ce2fe8-4a3f-4a02-a357-5e24226b12ec\") " Oct 30 02:06:39.601413 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:39.601243 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5ce2fe8-4a3f-4a02-a357-5e24226b12ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5ce2fe8-4a3f-4a02-a357-5e24226b12ec-plugins-config-volume\") pod \"d5ce2fe8-4a3f-4a02-a357-5e24226b12ec\" (UID: \"d5ce2fe8-4a3f-4a02-a357-5e24226b12ec\") " Oct 30 02:06:39.601413 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:39.601295 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5ce2fe8-4a3f-4a02-a357-5e24226b12ec-kube-api-access-jlrhd\" (UniqueName: \"kubernetes.io/projected/d5ce2fe8-4a3f-4a02-a357-5e24226b12ec-kube-api-access-jlrhd\") pod \"d5ce2fe8-4a3f-4a02-a357-5e24226b12ec\" (UID: \"d5ce2fe8-4a3f-4a02-a357-5e24226b12ec\") " Oct 30 02:06:39.601413 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:39.601381 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5ce2fe8-4a3f-4a02-a357-5e24226b12ec-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d5ce2fe8-4a3f-4a02-a357-5e24226b12ec-provider-credentials\") pod \"d5ce2fe8-4a3f-4a02-a357-5e24226b12ec\" (UID: \"d5ce2fe8-4a3f-4a02-a357-5e24226b12ec\") " Oct 30 02:06:39.601696 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:39.601664 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5ce2fe8-4a3f-4a02-a357-5e24226b12ec-routing-overlay-volume" pod "d5ce2fe8-4a3f-4a02-a357-5e24226b12ec" (UID: "d5ce2fe8-4a3f-4a02-a357-5e24226b12ec"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:06:39.601841 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:39.601726 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5ce2fe8-4a3f-4a02-a357-5e24226b12ec-plugins-config-volume" pod "d5ce2fe8-4a3f-4a02-a357-5e24226b12ec" (UID: "d5ce2fe8-4a3f-4a02-a357-5e24226b12ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:06:39.602191 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:39.602124 6221 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d5ce2fe8-4a3f-4a02-a357-5e24226b12ec-routing-overlay-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:06:39.602191 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:39.602150 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5ce2fe8-4a3f-4a02-a357-5e24226b12ec-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:06:39.603697 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:39.603676 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5ce2fe8-4a3f-4a02-a357-5e24226b12ec-kube-api-access-jlrhd" pod "d5ce2fe8-4a3f-4a02-a357-5e24226b12ec" (UID: "d5ce2fe8-4a3f-4a02-a357-5e24226b12ec"). InnerVolumeSpecName "kube-api-access-jlrhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:39.603784 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:39.603739 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5ce2fe8-4a3f-4a02-a357-5e24226b12ec-provider-credentials" pod "d5ce2fe8-4a3f-4a02-a357-5e24226b12ec" (UID: "d5ce2fe8-4a3f-4a02-a357-5e24226b12ec"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:39.703841 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:39.703806 6221 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d5ce2fe8-4a3f-4a02-a357-5e24226b12ec-provider-credentials\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:06:39.703841 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:39.703838 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlrhd\" (UniqueName: \"kubernetes.io/projected/d5ce2fe8-4a3f-4a02-a357-5e24226b12ec-kube-api-access-jlrhd\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:06:39.821041 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:39.821003 6221 generic.go:356] "Generic (PLEG): container finished" podID="d5ce2fe8-4a3f-4a02-a357-5e24226b12ec" containerID="7b803ca3c7db95470e9a55e3eea740093a2ac5ee286d1862a63c49ae210ca4dd" exitCode=0 Oct 30 02:06:39.821261 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:39.821053 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f55r9vm" Oct 30 02:06:39.821261 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:39.821086 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f55r9vm" event={"ID":"d5ce2fe8-4a3f-4a02-a357-5e24226b12ec","Type":"ContainerDied","Data":"7b803ca3c7db95470e9a55e3eea740093a2ac5ee286d1862a63c49ae210ca4dd"} Oct 30 02:06:39.821261 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:39.821147 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f55r9vm" event={"ID":"d5ce2fe8-4a3f-4a02-a357-5e24226b12ec","Type":"ContainerDied","Data":"9f9f341a7a2a66cfd4efe11a87758dc9a229d8da8fe905c113eec44f977f2c0f"} Oct 30 02:06:39.821261 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:39.821170 6221 scope.go:122] "RemoveContainer" containerID="7b803ca3c7db95470e9a55e3eea740093a2ac5ee286d1862a63c49ae210ca4dd" Oct 30 02:06:39.832631 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:39.832581 6221 scope.go:122] "RemoveContainer" containerID="7b803ca3c7db95470e9a55e3eea740093a2ac5ee286d1862a63c49ae210ca4dd" Oct 30 02:06:39.832871 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:06:39.832847 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b803ca3c7db95470e9a55e3eea740093a2ac5ee286d1862a63c49ae210ca4dd\": container with ID starting with 7b803ca3c7db95470e9a55e3eea740093a2ac5ee286d1862a63c49ae210ca4dd not found: ID does not exist" containerID="7b803ca3c7db95470e9a55e3eea740093a2ac5ee286d1862a63c49ae210ca4dd" Oct 30 02:06:39.832929 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:39.832880 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b803ca3c7db95470e9a55e3eea740093a2ac5ee286d1862a63c49ae210ca4dd"} err="failed to get container status \"7b803ca3c7db95470e9a55e3eea740093a2ac5ee286d1862a63c49ae210ca4dd\": rpc error: code = NotFound desc = could not find container \"7b803ca3c7db95470e9a55e3eea740093a2ac5ee286d1862a63c49ae210ca4dd\": container with ID starting with 7b803ca3c7db95470e9a55e3eea740093a2ac5ee286d1862a63c49ae210ca4dd not found: ID does not exist" Oct 30 02:06:39.874280 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:39.874245 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f55r9vm"] Oct 30 02:06:39.879135 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:39.879104 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-76a19e/payload-processing-external-model-e2e-extmod-a-76a19e-77f55r9vm"] Oct 30 02:06:40.213315 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:40.213212 6221 patch_prober.go:29] interesting pod/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:06:40.213315 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:40.213275 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5" podUID="23246593-d27c-4d51-a771-ae5b61df6fbb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:06:40.550666 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:40.550632 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5ce2fe8-4a3f-4a02-a357-5e24226b12ec" path="/var/lib/kubelet/pods/d5ce2fe8-4a3f-4a02-a357-5e24226b12ec/volumes" Oct 30 02:06:43.354868 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:43.354839 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5" Oct 30 02:06:43.484952 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:43.484901 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23246593-d27c-4d51-a771-ae5b61df6fbb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23246593-d27c-4d51-a771-ae5b61df6fbb-istiod-ca-cert\") pod \"23246593-d27c-4d51-a771-ae5b61df6fbb\" (UID: \"23246593-d27c-4d51-a771-ae5b61df6fbb\") " Oct 30 02:06:43.484952 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:43.484952 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23246593-d27c-4d51-a771-ae5b61df6fbb-kube-api-access-r2wtc\" (UniqueName: \"kubernetes.io/projected/23246593-d27c-4d51-a771-ae5b61df6fbb-kube-api-access-r2wtc\") pod \"23246593-d27c-4d51-a771-ae5b61df6fbb\" (UID: \"23246593-d27c-4d51-a771-ae5b61df6fbb\") " Oct 30 02:06:43.485243 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:43.485003 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23246593-d27c-4d51-a771-ae5b61df6fbb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23246593-d27c-4d51-a771-ae5b61df6fbb-istio-data\") pod \"23246593-d27c-4d51-a771-ae5b61df6fbb\" (UID: \"23246593-d27c-4d51-a771-ae5b61df6fbb\") " Oct 30 02:06:43.485243 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:43.485035 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23246593-d27c-4d51-a771-ae5b61df6fbb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23246593-d27c-4d51-a771-ae5b61df6fbb-workload-socket\") pod \"23246593-d27c-4d51-a771-ae5b61df6fbb\" (UID: \"23246593-d27c-4d51-a771-ae5b61df6fbb\") " Oct 30 02:06:43.485243 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:43.485091 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/23246593-d27c-4d51-a771-ae5b61df6fbb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23246593-d27c-4d51-a771-ae5b61df6fbb-istio-podinfo\") pod \"23246593-d27c-4d51-a771-ae5b61df6fbb\" (UID: \"23246593-d27c-4d51-a771-ae5b61df6fbb\") " Oct 30 02:06:43.485243 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:43.485141 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23246593-d27c-4d51-a771-ae5b61df6fbb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23246593-d27c-4d51-a771-ae5b61df6fbb-istio-envoy\") pod \"23246593-d27c-4d51-a771-ae5b61df6fbb\" (UID: \"23246593-d27c-4d51-a771-ae5b61df6fbb\") " Oct 30 02:06:43.485243 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:43.485181 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23246593-d27c-4d51-a771-ae5b61df6fbb-istio-token\" (UniqueName: \"kubernetes.io/projected/23246593-d27c-4d51-a771-ae5b61df6fbb-istio-token\") pod \"23246593-d27c-4d51-a771-ae5b61df6fbb\" (UID: \"23246593-d27c-4d51-a771-ae5b61df6fbb\") " Oct 30 02:06:43.485243 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:43.485242 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23246593-d27c-4d51-a771-ae5b61df6fbb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23246593-d27c-4d51-a771-ae5b61df6fbb-workload-certs\") pod \"23246593-d27c-4d51-a771-ae5b61df6fbb\" (UID: \"23246593-d27c-4d51-a771-ae5b61df6fbb\") " Oct 30 02:06:43.485614 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:43.485347 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23246593-d27c-4d51-a771-ae5b61df6fbb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23246593-d27c-4d51-a771-ae5b61df6fbb-credential-socket\") pod \"23246593-d27c-4d51-a771-ae5b61df6fbb\" (UID: \"23246593-d27c-4d51-a771-ae5b61df6fbb\") " Oct 30 02:06:43.485614 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:43.485406 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23246593-d27c-4d51-a771-ae5b61df6fbb-istio-data" pod "23246593-d27c-4d51-a771-ae5b61df6fbb" (UID: "23246593-d27c-4d51-a771-ae5b61df6fbb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:06:43.485614 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:43.485438 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23246593-d27c-4d51-a771-ae5b61df6fbb-istiod-ca-cert" pod "23246593-d27c-4d51-a771-ae5b61df6fbb" (UID: "23246593-d27c-4d51-a771-ae5b61df6fbb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:06:43.485782 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:43.485680 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23246593-d27c-4d51-a771-ae5b61df6fbb-workload-certs" pod "23246593-d27c-4d51-a771-ae5b61df6fbb" (UID: "23246593-d27c-4d51-a771-ae5b61df6fbb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:06:43.485782 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:43.485736 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23246593-d27c-4d51-a771-ae5b61df6fbb-workload-socket" pod "23246593-d27c-4d51-a771-ae5b61df6fbb" (UID: "23246593-d27c-4d51-a771-ae5b61df6fbb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:06:43.486114 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:43.486033 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23246593-d27c-4d51-a771-ae5b61df6fbb-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:06:43.486114 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:43.486056 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23246593-d27c-4d51-a771-ae5b61df6fbb-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:06:43.486114 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:43.486070 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23246593-d27c-4d51-a771-ae5b61df6fbb-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:06:43.486382 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:43.486170 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23246593-d27c-4d51-a771-ae5b61df6fbb-credential-socket" pod "23246593-d27c-4d51-a771-ae5b61df6fbb" (UID: "23246593-d27c-4d51-a771-ae5b61df6fbb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:06:43.486495 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:43.486468 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23246593-d27c-4d51-a771-ae5b61df6fbb-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:06:43.490661 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:43.488658 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23246593-d27c-4d51-a771-ae5b61df6fbb-istio-podinfo" pod "23246593-d27c-4d51-a771-ae5b61df6fbb" (UID: "23246593-d27c-4d51-a771-ae5b61df6fbb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:06:43.490661 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:43.488673 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23246593-d27c-4d51-a771-ae5b61df6fbb-istio-envoy" pod "23246593-d27c-4d51-a771-ae5b61df6fbb" (UID: "23246593-d27c-4d51-a771-ae5b61df6fbb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:06:43.490661 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:43.488743 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23246593-d27c-4d51-a771-ae5b61df6fbb-kube-api-access-r2wtc" pod "23246593-d27c-4d51-a771-ae5b61df6fbb" (UID: "23246593-d27c-4d51-a771-ae5b61df6fbb"). InnerVolumeSpecName "kube-api-access-r2wtc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:43.490661 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:43.488781 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23246593-d27c-4d51-a771-ae5b61df6fbb-istio-token" pod "23246593-d27c-4d51-a771-ae5b61df6fbb" (UID: "23246593-d27c-4d51-a771-ae5b61df6fbb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:43.592336 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:43.592218 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23246593-d27c-4d51-a771-ae5b61df6fbb-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:06:43.592336 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:43.592258 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23246593-d27c-4d51-a771-ae5b61df6fbb-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:06:43.592336 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:43.592271 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23246593-d27c-4d51-a771-ae5b61df6fbb-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:06:43.592336 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:43.592283 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23246593-d27c-4d51-a771-ae5b61df6fbb-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:06:43.592336 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:43.592298 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2wtc\" (UniqueName: \"kubernetes.io/projected/23246593-d27c-4d51-a771-ae5b61df6fbb-kube-api-access-r2wtc\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:06:43.687322 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:43.687281 6221 patch_prober.go:29] interesting pod/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:06:43.687544 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:43.687346 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" podUID="56476948-dc26-41d2-9df4-e23f8a293d75" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:06:43.858372 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:43.858335 6221 generic.go:356] "Generic (PLEG): container finished" podID="23246593-d27c-4d51-a771-ae5b61df6fbb" containerID="341ea0673c6deb48b8411452f7544c7454366302f9c16bfa3c92d9cccb0fe71d" exitCode=0 Oct 30 02:06:43.858583 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:43.858387 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5" Oct 30 02:06:43.858583 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:43.858421 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5" event={"ID":"23246593-d27c-4d51-a771-ae5b61df6fbb","Type":"ContainerDied","Data":"341ea0673c6deb48b8411452f7544c7454366302f9c16bfa3c92d9cccb0fe71d"} Oct 30 02:06:43.858583 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:43.858475 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5" event={"ID":"23246593-d27c-4d51-a771-ae5b61df6fbb","Type":"ContainerDied","Data":"042737ca5999f38f768fb329f501ae50435718bc1f97b296efd80b4fbe3d05f3"} Oct 30 02:06:43.858583 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:43.858498 6221 scope.go:122] "RemoveContainer" containerID="341ea0673c6deb48b8411452f7544c7454366302f9c16bfa3c92d9cccb0fe71d" Oct 30 02:06:43.869849 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:43.869826 6221 scope.go:122] "RemoveContainer" containerID="341ea0673c6deb48b8411452f7544c7454366302f9c16bfa3c92d9cccb0fe71d" Oct 30 02:06:43.870152 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:06:43.870131 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"341ea0673c6deb48b8411452f7544c7454366302f9c16bfa3c92d9cccb0fe71d\": container with ID starting with 341ea0673c6deb48b8411452f7544c7454366302f9c16bfa3c92d9cccb0fe71d not found: ID does not exist" containerID="341ea0673c6deb48b8411452f7544c7454366302f9c16bfa3c92d9cccb0fe71d" Oct 30 02:06:43.870231 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:43.870161 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"341ea0673c6deb48b8411452f7544c7454366302f9c16bfa3c92d9cccb0fe71d"} err="failed to get container status \"341ea0673c6deb48b8411452f7544c7454366302f9c16bfa3c92d9cccb0fe71d\": rpc error: code = NotFound desc = could not find container \"341ea0673c6deb48b8411452f7544c7454366302f9c16bfa3c92d9cccb0fe71d\": container with ID starting with 341ea0673c6deb48b8411452f7544c7454366302f9c16bfa3c92d9cccb0fe71d not found: ID does not exist" Oct 30 02:06:43.914653 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:43.914621 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5"] Oct 30 02:06:43.917890 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:43.917868 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-eda881-openshift-default-54f8644867-6djj5"] Oct 30 02:06:44.539351 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:06:44.539314 6221 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56476948_dc26_41d2_9df4_e23f8a293d75.slice/crio-conmon-77b25c8d03f17e9acf7d3fbe75a82ebf5d7a1a630fbd5420d9e588dcea5a5f01.scope\": RecentStats: unable to find data in memory cache]" Oct 30 02:06:44.551830 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.551801 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23246593-d27c-4d51-a771-ae5b61df6fbb" path="/var/lib/kubelet/pods/23246593-d27c-4d51-a771-ae5b61df6fbb/volumes" Oct 30 02:06:44.560836 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.560806 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" Oct 30 02:06:44.606490 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.604985 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48557095-c2dc-40cb-8ee3-8ee71567c61c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48557095-c2dc-40cb-8ee3-8ee71567c61c-workload-certs\") pod \"48557095-c2dc-40cb-8ee3-8ee71567c61c\" (UID: \"48557095-c2dc-40cb-8ee3-8ee71567c61c\") " Oct 30 02:06:44.606490 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.605047 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48557095-c2dc-40cb-8ee3-8ee71567c61c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48557095-c2dc-40cb-8ee3-8ee71567c61c-istiod-ca-cert\") pod \"48557095-c2dc-40cb-8ee3-8ee71567c61c\" (UID: \"48557095-c2dc-40cb-8ee3-8ee71567c61c\") " Oct 30 02:06:44.606490 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.605154 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48557095-c2dc-40cb-8ee3-8ee71567c61c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48557095-c2dc-40cb-8ee3-8ee71567c61c-credential-socket\") pod \"48557095-c2dc-40cb-8ee3-8ee71567c61c\" (UID: \"48557095-c2dc-40cb-8ee3-8ee71567c61c\") " Oct 30 02:06:44.606490 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.605223 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48557095-c2dc-40cb-8ee3-8ee71567c61c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48557095-c2dc-40cb-8ee3-8ee71567c61c-istio-envoy\") pod \"48557095-c2dc-40cb-8ee3-8ee71567c61c\" (UID: \"48557095-c2dc-40cb-8ee3-8ee71567c61c\") " Oct 30 02:06:44.606490 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.605284 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48557095-c2dc-40cb-8ee3-8ee71567c61c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48557095-c2dc-40cb-8ee3-8ee71567c61c-istio-podinfo\") pod \"48557095-c2dc-40cb-8ee3-8ee71567c61c\" (UID: \"48557095-c2dc-40cb-8ee3-8ee71567c61c\") " Oct 30 02:06:44.606490 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.605316 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48557095-c2dc-40cb-8ee3-8ee71567c61c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48557095-c2dc-40cb-8ee3-8ee71567c61c-istio-data\") pod \"48557095-c2dc-40cb-8ee3-8ee71567c61c\" (UID: \"48557095-c2dc-40cb-8ee3-8ee71567c61c\") " Oct 30 02:06:44.606490 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.605350 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48557095-c2dc-40cb-8ee3-8ee71567c61c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48557095-c2dc-40cb-8ee3-8ee71567c61c-workload-socket\") pod \"48557095-c2dc-40cb-8ee3-8ee71567c61c\" (UID: \"48557095-c2dc-40cb-8ee3-8ee71567c61c\") " Oct 30 02:06:44.606490 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.605391 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48557095-c2dc-40cb-8ee3-8ee71567c61c-istio-token\" (UniqueName: \"kubernetes.io/projected/48557095-c2dc-40cb-8ee3-8ee71567c61c-istio-token\") pod \"48557095-c2dc-40cb-8ee3-8ee71567c61c\" (UID: \"48557095-c2dc-40cb-8ee3-8ee71567c61c\") " Oct 30 02:06:44.609924 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.609880 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48557095-c2dc-40cb-8ee3-8ee71567c61c-workload-certs" pod "48557095-c2dc-40cb-8ee3-8ee71567c61c" (UID: "48557095-c2dc-40cb-8ee3-8ee71567c61c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:06:44.616611 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.612108 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48557095-c2dc-40cb-8ee3-8ee71567c61c-kube-api-access-4jbts\" (UniqueName: \"kubernetes.io/projected/48557095-c2dc-40cb-8ee3-8ee71567c61c-kube-api-access-4jbts\") pod \"48557095-c2dc-40cb-8ee3-8ee71567c61c\" (UID: \"48557095-c2dc-40cb-8ee3-8ee71567c61c\") " Oct 30 02:06:44.616611 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.613079 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48557095-c2dc-40cb-8ee3-8ee71567c61c-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:06:44.619834 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.617700 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48557095-c2dc-40cb-8ee3-8ee71567c61c-kube-api-access-4jbts" pod "48557095-c2dc-40cb-8ee3-8ee71567c61c" (UID: "48557095-c2dc-40cb-8ee3-8ee71567c61c"). InnerVolumeSpecName "kube-api-access-4jbts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:44.619834 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.617829 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48557095-c2dc-40cb-8ee3-8ee71567c61c-istiod-ca-cert" pod "48557095-c2dc-40cb-8ee3-8ee71567c61c" (UID: "48557095-c2dc-40cb-8ee3-8ee71567c61c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:06:44.619834 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.618920 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48557095-c2dc-40cb-8ee3-8ee71567c61c-istio-podinfo" pod "48557095-c2dc-40cb-8ee3-8ee71567c61c" (UID: "48557095-c2dc-40cb-8ee3-8ee71567c61c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:06:44.632441 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.627415 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48557095-c2dc-40cb-8ee3-8ee71567c61c-workload-socket" pod "48557095-c2dc-40cb-8ee3-8ee71567c61c" (UID: "48557095-c2dc-40cb-8ee3-8ee71567c61c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:06:44.632441 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.628666 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48557095-c2dc-40cb-8ee3-8ee71567c61c-credential-socket" pod "48557095-c2dc-40cb-8ee3-8ee71567c61c" (UID: "48557095-c2dc-40cb-8ee3-8ee71567c61c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:06:44.632441 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.628754 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48557095-c2dc-40cb-8ee3-8ee71567c61c-istio-data" pod "48557095-c2dc-40cb-8ee3-8ee71567c61c" (UID: "48557095-c2dc-40cb-8ee3-8ee71567c61c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:06:44.632441 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.632335 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48557095-c2dc-40cb-8ee3-8ee71567c61c-istio-token" pod "48557095-c2dc-40cb-8ee3-8ee71567c61c" (UID: "48557095-c2dc-40cb-8ee3-8ee71567c61c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:44.641755 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.635411 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48557095-c2dc-40cb-8ee3-8ee71567c61c-istio-envoy" pod "48557095-c2dc-40cb-8ee3-8ee71567c61c" (UID: "48557095-c2dc-40cb-8ee3-8ee71567c61c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:06:44.728709 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.728621 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48557095-c2dc-40cb-8ee3-8ee71567c61c-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:06:44.728709 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.728660 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48557095-c2dc-40cb-8ee3-8ee71567c61c-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:06:44.728709 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.728674 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48557095-c2dc-40cb-8ee3-8ee71567c61c-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:06:44.728709 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.728686 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48557095-c2dc-40cb-8ee3-8ee71567c61c-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:06:44.728709 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.728698 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48557095-c2dc-40cb-8ee3-8ee71567c61c-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:06:44.728709 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.728710 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48557095-c2dc-40cb-8ee3-8ee71567c61c-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:06:44.729179 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.728728 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jbts\" (UniqueName: \"kubernetes.io/projected/48557095-c2dc-40cb-8ee3-8ee71567c61c-kube-api-access-4jbts\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:06:44.729179 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.728745 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48557095-c2dc-40cb-8ee3-8ee71567c61c-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:06:44.743128 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.743102 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" Oct 30 02:06:44.829627 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.829594 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56476948-dc26-41d2-9df4-e23f8a293d75-istio-token\" (UniqueName: \"kubernetes.io/projected/56476948-dc26-41d2-9df4-e23f8a293d75-istio-token\") pod \"56476948-dc26-41d2-9df4-e23f8a293d75\" (UID: \"56476948-dc26-41d2-9df4-e23f8a293d75\") " Oct 30 02:06:44.829844 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.829642 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56476948-dc26-41d2-9df4-e23f8a293d75-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56476948-dc26-41d2-9df4-e23f8a293d75-credential-socket\") pod \"56476948-dc26-41d2-9df4-e23f8a293d75\" (UID: \"56476948-dc26-41d2-9df4-e23f8a293d75\") " Oct 30 02:06:44.829844 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.829729 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56476948-dc26-41d2-9df4-e23f8a293d75-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56476948-dc26-41d2-9df4-e23f8a293d75-workload-certs\") pod \"56476948-dc26-41d2-9df4-e23f8a293d75\" (UID: \"56476948-dc26-41d2-9df4-e23f8a293d75\") " Oct 30 02:06:44.829844 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.829782 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56476948-dc26-41d2-9df4-e23f8a293d75-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56476948-dc26-41d2-9df4-e23f8a293d75-istiod-ca-cert\") pod \"56476948-dc26-41d2-9df4-e23f8a293d75\" (UID: \"56476948-dc26-41d2-9df4-e23f8a293d75\") " Oct 30 02:06:44.829844 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.829820 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56476948-dc26-41d2-9df4-e23f8a293d75-kube-api-access-rwc7f\" (UniqueName: \"kubernetes.io/projected/56476948-dc26-41d2-9df4-e23f8a293d75-kube-api-access-rwc7f\") pod \"56476948-dc26-41d2-9df4-e23f8a293d75\" (UID: \"56476948-dc26-41d2-9df4-e23f8a293d75\") " Oct 30 02:06:44.830055 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.829926 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56476948-dc26-41d2-9df4-e23f8a293d75-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56476948-dc26-41d2-9df4-e23f8a293d75-istio-envoy\") pod \"56476948-dc26-41d2-9df4-e23f8a293d75\" (UID: \"56476948-dc26-41d2-9df4-e23f8a293d75\") " Oct 30 02:06:44.830055 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.829982 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56476948-dc26-41d2-9df4-e23f8a293d75-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56476948-dc26-41d2-9df4-e23f8a293d75-workload-socket\") pod \"56476948-dc26-41d2-9df4-e23f8a293d75\" (UID: \"56476948-dc26-41d2-9df4-e23f8a293d75\") " Oct 30 02:06:44.830055 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.830008 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56476948-dc26-41d2-9df4-e23f8a293d75-workload-certs" pod "56476948-dc26-41d2-9df4-e23f8a293d75" (UID: "56476948-dc26-41d2-9df4-e23f8a293d75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:06:44.830212 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.830160 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56476948-dc26-41d2-9df4-e23f8a293d75-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56476948-dc26-41d2-9df4-e23f8a293d75-istio-data\") pod \"56476948-dc26-41d2-9df4-e23f8a293d75\" (UID: \"56476948-dc26-41d2-9df4-e23f8a293d75\") " Oct 30 02:06:44.830212 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.830167 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56476948-dc26-41d2-9df4-e23f8a293d75-credential-socket" pod "56476948-dc26-41d2-9df4-e23f8a293d75" (UID: "56476948-dc26-41d2-9df4-e23f8a293d75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:06:44.830212 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.830197 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56476948-dc26-41d2-9df4-e23f8a293d75-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56476948-dc26-41d2-9df4-e23f8a293d75-istio-podinfo\") pod \"56476948-dc26-41d2-9df4-e23f8a293d75\" (UID: \"56476948-dc26-41d2-9df4-e23f8a293d75\") " Oct 30 02:06:44.830549 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.830513 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56476948-dc26-41d2-9df4-e23f8a293d75-workload-socket" pod "56476948-dc26-41d2-9df4-e23f8a293d75" (UID: "56476948-dc26-41d2-9df4-e23f8a293d75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:06:44.830904 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.830881 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56476948-dc26-41d2-9df4-e23f8a293d75-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:06:44.831057 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.830907 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56476948-dc26-41d2-9df4-e23f8a293d75-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:06:44.831057 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.830922 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56476948-dc26-41d2-9df4-e23f8a293d75-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:06:44.831271 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.831146 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56476948-dc26-41d2-9df4-e23f8a293d75-istio-data" pod "56476948-dc26-41d2-9df4-e23f8a293d75" (UID: "56476948-dc26-41d2-9df4-e23f8a293d75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:06:44.831415 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.831391 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56476948-dc26-41d2-9df4-e23f8a293d75-istiod-ca-cert" pod "56476948-dc26-41d2-9df4-e23f8a293d75" (UID: "56476948-dc26-41d2-9df4-e23f8a293d75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:06:44.833028 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.832997 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56476948-dc26-41d2-9df4-e23f8a293d75-istio-envoy" pod "56476948-dc26-41d2-9df4-e23f8a293d75" (UID: "56476948-dc26-41d2-9df4-e23f8a293d75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:06:44.833398 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.833369 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56476948-dc26-41d2-9df4-e23f8a293d75-istio-podinfo" pod "56476948-dc26-41d2-9df4-e23f8a293d75" (UID: "56476948-dc26-41d2-9df4-e23f8a293d75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:06:44.833494 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.833391 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56476948-dc26-41d2-9df4-e23f8a293d75-kube-api-access-rwc7f" pod "56476948-dc26-41d2-9df4-e23f8a293d75" (UID: "56476948-dc26-41d2-9df4-e23f8a293d75"). InnerVolumeSpecName "kube-api-access-rwc7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:44.833494 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.833437 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56476948-dc26-41d2-9df4-e23f8a293d75-istio-token" pod "56476948-dc26-41d2-9df4-e23f8a293d75" (UID: "56476948-dc26-41d2-9df4-e23f8a293d75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:44.867923 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.867886 6221 generic.go:356] "Generic (PLEG): container finished" podID="48557095-c2dc-40cb-8ee3-8ee71567c61c" containerID="02b28bf6299d286f6ec9a31d31008878b76c5e36c7bafa1122a21fa124e1219f" exitCode=0 Oct 30 02:06:44.868094 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.867940 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" Oct 30 02:06:44.868094 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.867958 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" event={"ID":"48557095-c2dc-40cb-8ee3-8ee71567c61c","Type":"ContainerDied","Data":"02b28bf6299d286f6ec9a31d31008878b76c5e36c7bafa1122a21fa124e1219f"} Oct 30 02:06:44.868094 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.868028 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9" event={"ID":"48557095-c2dc-40cb-8ee3-8ee71567c61c","Type":"ContainerDied","Data":"ab50521be4f8cc8990d422aa7453f027e6088e5bd94e3d3619c43a2ba8bbf22a"} Oct 30 02:06:44.868094 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.868057 6221 scope.go:122] "RemoveContainer" containerID="02b28bf6299d286f6ec9a31d31008878b76c5e36c7bafa1122a21fa124e1219f" Oct 30 02:06:44.870054 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.870024 6221 generic.go:356] "Generic (PLEG): container finished" podID="56476948-dc26-41d2-9df4-e23f8a293d75" containerID="77b25c8d03f17e9acf7d3fbe75a82ebf5d7a1a630fbd5420d9e588dcea5a5f01" exitCode=0 Oct 30 02:06:44.870178 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.870073 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" Oct 30 02:06:44.870178 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.870119 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" event={"ID":"56476948-dc26-41d2-9df4-e23f8a293d75","Type":"ContainerDied","Data":"77b25c8d03f17e9acf7d3fbe75a82ebf5d7a1a630fbd5420d9e588dcea5a5f01"} Oct 30 02:06:44.870178 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.870174 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d" event={"ID":"56476948-dc26-41d2-9df4-e23f8a293d75","Type":"ContainerDied","Data":"cfff332c899cf7186306218413a0d2e099ba7e61aaea6dc6a47b5b8fe8fe3f8a"} Oct 30 02:06:44.881606 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.881512 6221 scope.go:122] "RemoveContainer" containerID="02b28bf6299d286f6ec9a31d31008878b76c5e36c7bafa1122a21fa124e1219f" Oct 30 02:06:44.881898 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:06:44.881869 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02b28bf6299d286f6ec9a31d31008878b76c5e36c7bafa1122a21fa124e1219f\": container with ID starting with 02b28bf6299d286f6ec9a31d31008878b76c5e36c7bafa1122a21fa124e1219f not found: ID does not exist" containerID="02b28bf6299d286f6ec9a31d31008878b76c5e36c7bafa1122a21fa124e1219f" Oct 30 02:06:44.881974 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.881912 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02b28bf6299d286f6ec9a31d31008878b76c5e36c7bafa1122a21fa124e1219f"} err="failed to get container status \"02b28bf6299d286f6ec9a31d31008878b76c5e36c7bafa1122a21fa124e1219f\": rpc error: code = NotFound desc = could not find container \"02b28bf6299d286f6ec9a31d31008878b76c5e36c7bafa1122a21fa124e1219f\": container with ID starting with 02b28bf6299d286f6ec9a31d31008878b76c5e36c7bafa1122a21fa124e1219f not found: ID does not exist" Oct 30 02:06:44.881974 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.881937 6221 scope.go:122] "RemoveContainer" containerID="77b25c8d03f17e9acf7d3fbe75a82ebf5d7a1a630fbd5420d9e588dcea5a5f01" Oct 30 02:06:44.892346 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.892324 6221 scope.go:122] "RemoveContainer" containerID="e19c9fa6d3979b03b5ef652ad07285786196e7158988e3dd4c87217b0087e747" Oct 30 02:06:44.902524 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.902494 6221 scope.go:122] "RemoveContainer" containerID="77b25c8d03f17e9acf7d3fbe75a82ebf5d7a1a630fbd5420d9e588dcea5a5f01" Oct 30 02:06:44.902843 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:06:44.902824 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77b25c8d03f17e9acf7d3fbe75a82ebf5d7a1a630fbd5420d9e588dcea5a5f01\": container with ID starting with 77b25c8d03f17e9acf7d3fbe75a82ebf5d7a1a630fbd5420d9e588dcea5a5f01 not found: ID does not exist" containerID="77b25c8d03f17e9acf7d3fbe75a82ebf5d7a1a630fbd5420d9e588dcea5a5f01" Oct 30 02:06:44.902912 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.902851 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77b25c8d03f17e9acf7d3fbe75a82ebf5d7a1a630fbd5420d9e588dcea5a5f01"} err="failed to get container status \"77b25c8d03f17e9acf7d3fbe75a82ebf5d7a1a630fbd5420d9e588dcea5a5f01\": rpc error: code = NotFound desc = could not find container \"77b25c8d03f17e9acf7d3fbe75a82ebf5d7a1a630fbd5420d9e588dcea5a5f01\": container with ID starting with 77b25c8d03f17e9acf7d3fbe75a82ebf5d7a1a630fbd5420d9e588dcea5a5f01 not found: ID does not exist" Oct 30 02:06:44.902912 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.902868 6221 scope.go:122] "RemoveContainer" containerID="e19c9fa6d3979b03b5ef652ad07285786196e7158988e3dd4c87217b0087e747" Oct 30 02:06:44.903112 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:06:44.903099 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e19c9fa6d3979b03b5ef652ad07285786196e7158988e3dd4c87217b0087e747\": container with ID starting with e19c9fa6d3979b03b5ef652ad07285786196e7158988e3dd4c87217b0087e747 not found: ID does not exist" containerID="e19c9fa6d3979b03b5ef652ad07285786196e7158988e3dd4c87217b0087e747" Oct 30 02:06:44.903155 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.903115 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e19c9fa6d3979b03b5ef652ad07285786196e7158988e3dd4c87217b0087e747"} err="failed to get container status \"e19c9fa6d3979b03b5ef652ad07285786196e7158988e3dd4c87217b0087e747\": rpc error: code = NotFound desc = could not find container \"e19c9fa6d3979b03b5ef652ad07285786196e7158988e3dd4c87217b0087e747\": container with ID starting with e19c9fa6d3979b03b5ef652ad07285786196e7158988e3dd4c87217b0087e747 not found: ID does not exist" Oct 30 02:06:44.935554 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.932976 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d"] Oct 30 02:06:44.935554 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.934442 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56476948-dc26-41d2-9df4-e23f8a293d75-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:06:44.935554 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.934472 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56476948-dc26-41d2-9df4-e23f8a293d75-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:06:44.935554 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.934486 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56476948-dc26-41d2-9df4-e23f8a293d75-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:06:44.935554 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.934501 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56476948-dc26-41d2-9df4-e23f8a293d75-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:06:44.935554 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.934514 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56476948-dc26-41d2-9df4-e23f8a293d75-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:06:44.935554 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:44.934543 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwc7f\" (UniqueName: \"kubernetes.io/projected/56476948-dc26-41d2-9df4-e23f8a293d75-kube-api-access-rwc7f\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:06:45.018711 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:45.017079 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-657fba-openshift-default-747b8794ff-vt76d"] Oct 30 02:06:45.052844 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:45.052658 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9"] Oct 30 02:06:45.057594 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:45.057267 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a3a2ac-openshift-default-7cc4bd87f5-d59n9"] Oct 30 02:06:45.233336 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:45.233304 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-666655759c-gvdcm"] Oct 30 02:06:45.234385 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:45.234361 6221 cpu_manager.go:414] "containerMap: removing container" podUID="23246593-d27c-4d51-a771-ae5b61df6fbb" containerName="istio-proxy" Oct 30 02:06:45.234509 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:45.234387 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23246593-d27c-4d51-a771-ae5b61df6fbb" containerName="istio-proxy" Oct 30 02:06:45.234509 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:45.234402 6221 cpu_manager.go:414] "containerMap: removing container" podUID="56476948-dc26-41d2-9df4-e23f8a293d75" containerName="istio-proxy" Oct 30 02:06:45.234509 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:45.234409 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56476948-dc26-41d2-9df4-e23f8a293d75" containerName="istio-proxy" Oct 30 02:06:45.234509 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:45.234440 6221 cpu_manager.go:414] "containerMap: removing container" podUID="d5ce2fe8-4a3f-4a02-a357-5e24226b12ec" containerName="payload-processing" Oct 30 02:06:45.234509 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:45.234448 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5ce2fe8-4a3f-4a02-a357-5e24226b12ec" containerName="payload-processing" Oct 30 02:06:45.234509 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:45.234485 6221 cpu_manager.go:414] "containerMap: removing container" podUID="48557095-c2dc-40cb-8ee3-8ee71567c61c" containerName="istio-proxy" Oct 30 02:06:45.234509 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:45.234490 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48557095-c2dc-40cb-8ee3-8ee71567c61c" containerName="istio-proxy" Oct 30 02:06:45.234795 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:45.234639 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="48557095-c2dc-40cb-8ee3-8ee71567c61c" containerName="istio-proxy" Oct 30 02:06:45.234795 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:45.234651 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="56476948-dc26-41d2-9df4-e23f8a293d75" containerName="istio-proxy" Oct 30 02:06:45.234795 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:45.234660 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="23246593-d27c-4d51-a771-ae5b61df6fbb" containerName="istio-proxy" Oct 30 02:06:45.234795 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:45.234666 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="56476948-dc26-41d2-9df4-e23f8a293d75" containerName="istio-proxy" Oct 30 02:06:45.234795 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:45.234673 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5ce2fe8-4a3f-4a02-a357-5e24226b12ec" containerName="payload-processing" Oct 30 02:06:45.239912 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:45.239893 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-666655759c-gvdcm" Oct 30 02:06:45.262109 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:45.262075 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-84d872aa7a07\"" Oct 30 02:06:45.262319 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:45.262266 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-7ecb73ededa6\"" Oct 30 02:06:45.262876 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:45.262844 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-nfgtq\"" Oct 30 02:06:45.265957 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:45.265932 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 02:06:45.266098 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:45.266039 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 02:06:45.266248 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:45.266229 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 02:06:45.266480 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:45.266330 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 02:06:45.276322 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:45.273801 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-666655759c-gvdcm"] Oct 30 02:06:45.343466 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:45.343423 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl68g\" (UniqueName: \"kubernetes.io/projected/79f3f095-838b-4342-860b-0bfef6ffddae-kube-api-access-gl68g\") pod \"payload-processing-external-model-666655759c-gvdcm\" (UID: \"79f3f095-838b-4342-860b-0bfef6ffddae\") " pod="models-as-a-service/payload-processing-external-model-666655759c-gvdcm" Oct 30 02:06:45.343697 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:45.343619 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/79f3f095-838b-4342-860b-0bfef6ffddae-provider-credentials\") pod \"payload-processing-external-model-666655759c-gvdcm\" (UID: \"79f3f095-838b-4342-860b-0bfef6ffddae\") " pod="models-as-a-service/payload-processing-external-model-666655759c-gvdcm" Oct 30 02:06:45.343767 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:45.343733 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/79f3f095-838b-4342-860b-0bfef6ffddae-routing-overlay-volume\") pod \"payload-processing-external-model-666655759c-gvdcm\" (UID: \"79f3f095-838b-4342-860b-0bfef6ffddae\") " pod="models-as-a-service/payload-processing-external-model-666655759c-gvdcm" Oct 30 02:06:45.343888 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:45.343862 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79f3f095-838b-4342-860b-0bfef6ffddae-plugins-config-volume\") pod \"payload-processing-external-model-666655759c-gvdcm\" (UID: \"79f3f095-838b-4342-860b-0bfef6ffddae\") " pod="models-as-a-service/payload-processing-external-model-666655759c-gvdcm" Oct 30 02:06:45.364465 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:45.364430 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh"] Oct 30 02:06:45.364709 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:45.364685 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh" podUID="59eaf4dd-3059-4fa3-bac8-93fc2b4d518a" containerName="istio-proxy" containerID="cri-o://7ed4f067ebd5c37303404d57b1d652dd61a80ce5364d9e8b2907dc535a0f954d" gracePeriod=30 Oct 30 02:06:45.445651 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:45.445608 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79f3f095-838b-4342-860b-0bfef6ffddae-plugins-config-volume\") pod \"payload-processing-external-model-666655759c-gvdcm\" (UID: \"79f3f095-838b-4342-860b-0bfef6ffddae\") " pod="models-as-a-service/payload-processing-external-model-666655759c-gvdcm" Oct 30 02:06:45.445868 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:45.445776 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl68g\" (UniqueName: \"kubernetes.io/projected/79f3f095-838b-4342-860b-0bfef6ffddae-kube-api-access-gl68g\") pod \"payload-processing-external-model-666655759c-gvdcm\" (UID: \"79f3f095-838b-4342-860b-0bfef6ffddae\") " pod="models-as-a-service/payload-processing-external-model-666655759c-gvdcm" Oct 30 02:06:45.445955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:45.445870 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/79f3f095-838b-4342-860b-0bfef6ffddae-provider-credentials\") pod \"payload-processing-external-model-666655759c-gvdcm\" (UID: \"79f3f095-838b-4342-860b-0bfef6ffddae\") " pod="models-as-a-service/payload-processing-external-model-666655759c-gvdcm" Oct 30 02:06:45.446024 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:45.445951 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/79f3f095-838b-4342-860b-0bfef6ffddae-routing-overlay-volume\") pod \"payload-processing-external-model-666655759c-gvdcm\" (UID: \"79f3f095-838b-4342-860b-0bfef6ffddae\") " pod="models-as-a-service/payload-processing-external-model-666655759c-gvdcm" Oct 30 02:06:45.446714 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:45.446683 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/79f3f095-838b-4342-860b-0bfef6ffddae-routing-overlay-volume\") pod \"payload-processing-external-model-666655759c-gvdcm\" (UID: \"79f3f095-838b-4342-860b-0bfef6ffddae\") " pod="models-as-a-service/payload-processing-external-model-666655759c-gvdcm" Oct 30 02:06:45.446835 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:45.446744 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79f3f095-838b-4342-860b-0bfef6ffddae-plugins-config-volume\") pod \"payload-processing-external-model-666655759c-gvdcm\" (UID: \"79f3f095-838b-4342-860b-0bfef6ffddae\") " pod="models-as-a-service/payload-processing-external-model-666655759c-gvdcm" Oct 30 02:06:45.449651 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:45.449620 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/79f3f095-838b-4342-860b-0bfef6ffddae-provider-credentials\") pod \"payload-processing-external-model-666655759c-gvdcm\" (UID: \"79f3f095-838b-4342-860b-0bfef6ffddae\") " pod="models-as-a-service/payload-processing-external-model-666655759c-gvdcm" Oct 30 02:06:45.466767 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:45.466734 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl68g\" (UniqueName: \"kubernetes.io/projected/79f3f095-838b-4342-860b-0bfef6ffddae-kube-api-access-gl68g\") pod \"payload-processing-external-model-666655759c-gvdcm\" (UID: \"79f3f095-838b-4342-860b-0bfef6ffddae\") " pod="models-as-a-service/payload-processing-external-model-666655759c-gvdcm" Oct 30 02:06:45.559847 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:45.559798 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-666655759c-gvdcm" Oct 30 02:06:45.981850 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:45.981820 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-666655759c-gvdcm"] Oct 30 02:06:45.984399 ip-10-0-135-65 kubenswrapper[6221]: W1030 02:06:45.984368 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79f3f095_838b_4342_860b_0bfef6ffddae.slice/crio-38e04ad49576532d2988067a3b645fb204cd13db0fa80084af97e920c27f520d WatchSource:0}: Error finding container 38e04ad49576532d2988067a3b645fb204cd13db0fa80084af97e920c27f520d: Status 404 returned error can't find the container with id 38e04ad49576532d2988067a3b645fb204cd13db0fa80084af97e920c27f520d Oct 30 02:06:46.550434 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:46.550397 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48557095-c2dc-40cb-8ee3-8ee71567c61c" path="/var/lib/kubelet/pods/48557095-c2dc-40cb-8ee3-8ee71567c61c/volumes" Oct 30 02:06:46.550944 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:46.550928 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56476948-dc26-41d2-9df4-e23f8a293d75" path="/var/lib/kubelet/pods/56476948-dc26-41d2-9df4-e23f8a293d75/volumes" Oct 30 02:06:46.891849 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:46.891760 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-666655759c-gvdcm" event={"ID":"79f3f095-838b-4342-860b-0bfef6ffddae","Type":"ContainerStarted","Data":"341c40c0e74c80b5e9fccd7bbd270d5be958fbd6d767876743a7600fe98b1482"} Oct 30 02:06:46.891849 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:46.891814 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-666655759c-gvdcm" event={"ID":"79f3f095-838b-4342-860b-0bfef6ffddae","Type":"ContainerStarted","Data":"38e04ad49576532d2988067a3b645fb204cd13db0fa80084af97e920c27f520d"} Oct 30 02:06:46.891849 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:46.891840 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-666655759c-gvdcm" Oct 30 02:06:46.942793 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:46.942720 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-666655759c-gvdcm" podStartSLOduration=1.942705106 podStartE2EDuration="1.942705106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:06:46.941590664 +0000 UTC m=+2930.519878000" watchObservedRunningTime="2026-10-30 02:06:46.942705106 +0000 UTC m=+2930.520992441" Oct 30 02:06:47.227609 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:47.226428 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m"] Oct 30 02:06:47.227800 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:47.227680 6221 cpu_manager.go:414] "containerMap: removing container" podUID="56476948-dc26-41d2-9df4-e23f8a293d75" containerName="istio-proxy" Oct 30 02:06:47.227800 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:47.227704 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56476948-dc26-41d2-9df4-e23f8a293d75" containerName="istio-proxy" Oct 30 02:06:47.234156 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:47.234124 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" Oct 30 02:06:47.270594 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:47.270558 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5227b83-d834-4276-bd61-27e1251a0708-workload-certs\") pod \"e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m\" (UID: \"a5227b83-d834-4276-bd61-27e1251a0708\") " pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" Oct 30 02:06:47.270793 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:47.270766 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5pq8\" (UniqueName: \"kubernetes.io/projected/a5227b83-d834-4276-bd61-27e1251a0708-kube-api-access-s5pq8\") pod \"e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m\" (UID: \"a5227b83-d834-4276-bd61-27e1251a0708\") " pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" Oct 30 02:06:47.270840 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:47.270815 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5227b83-d834-4276-bd61-27e1251a0708-istiod-ca-cert\") pod \"e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m\" (UID: \"a5227b83-d834-4276-bd61-27e1251a0708\") " pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" Oct 30 02:06:47.270916 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:47.270896 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5227b83-d834-4276-bd61-27e1251a0708-istio-data\") pod \"e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m\" (UID: \"a5227b83-d834-4276-bd61-27e1251a0708\") " pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" Oct 30 02:06:47.271001 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:47.270986 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5227b83-d834-4276-bd61-27e1251a0708-workload-socket\") pod \"e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m\" (UID: \"a5227b83-d834-4276-bd61-27e1251a0708\") " pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" Oct 30 02:06:47.271106 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:47.271015 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5227b83-d834-4276-bd61-27e1251a0708-istio-envoy\") pod \"e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m\" (UID: \"a5227b83-d834-4276-bd61-27e1251a0708\") " pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" Oct 30 02:06:47.271106 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:47.271039 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5227b83-d834-4276-bd61-27e1251a0708-credential-socket\") pod \"e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m\" (UID: \"a5227b83-d834-4276-bd61-27e1251a0708\") " pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" Oct 30 02:06:47.271187 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:47.271098 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5227b83-d834-4276-bd61-27e1251a0708-istio-token\") pod \"e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m\" (UID: \"a5227b83-d834-4276-bd61-27e1251a0708\") " pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" Oct 30 02:06:47.271187 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:47.271168 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5227b83-d834-4276-bd61-27e1251a0708-istio-podinfo\") pod \"e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m\" (UID: \"a5227b83-d834-4276-bd61-27e1251a0708\") " pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" Oct 30 02:06:47.298940 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:47.298903 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-aaf045-openshift-default-dockercfg-lhnx4\"" Oct 30 02:06:47.318833 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:47.318792 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m"] Oct 30 02:06:47.385429 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:47.385324 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5227b83-d834-4276-bd61-27e1251a0708-workload-socket\") pod \"e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m\" (UID: \"a5227b83-d834-4276-bd61-27e1251a0708\") " pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" Oct 30 02:06:47.385691 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:47.385452 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5227b83-d834-4276-bd61-27e1251a0708-istio-envoy\") pod \"e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m\" (UID: \"a5227b83-d834-4276-bd61-27e1251a0708\") " pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" Oct 30 02:06:47.385691 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:47.385483 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5227b83-d834-4276-bd61-27e1251a0708-credential-socket\") pod \"e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m\" (UID: \"a5227b83-d834-4276-bd61-27e1251a0708\") " pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" Oct 30 02:06:47.385691 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:47.385516 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5227b83-d834-4276-bd61-27e1251a0708-istio-token\") pod \"e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m\" (UID: \"a5227b83-d834-4276-bd61-27e1251a0708\") " pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" Oct 30 02:06:47.385691 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:47.385592 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5227b83-d834-4276-bd61-27e1251a0708-istio-podinfo\") pod \"e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m\" (UID: \"a5227b83-d834-4276-bd61-27e1251a0708\") " pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" Oct 30 02:06:47.385691 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:47.385684 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5227b83-d834-4276-bd61-27e1251a0708-workload-certs\") pod \"e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m\" (UID: \"a5227b83-d834-4276-bd61-27e1251a0708\") " pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" Oct 30 02:06:47.385992 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:47.385876 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5pq8\" (UniqueName: \"kubernetes.io/projected/a5227b83-d834-4276-bd61-27e1251a0708-kube-api-access-s5pq8\") pod \"e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m\" (UID: \"a5227b83-d834-4276-bd61-27e1251a0708\") " pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" Oct 30 02:06:47.385992 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:47.385923 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5227b83-d834-4276-bd61-27e1251a0708-istiod-ca-cert\") pod \"e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m\" (UID: \"a5227b83-d834-4276-bd61-27e1251a0708\") " pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" Oct 30 02:06:47.385992 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:47.385960 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5227b83-d834-4276-bd61-27e1251a0708-istio-data\") pod \"e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m\" (UID: \"a5227b83-d834-4276-bd61-27e1251a0708\") " pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" Oct 30 02:06:47.386558 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:47.386453 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5227b83-d834-4276-bd61-27e1251a0708-istio-data\") pod \"e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m\" (UID: \"a5227b83-d834-4276-bd61-27e1251a0708\") " pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" Oct 30 02:06:47.387004 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:47.386973 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5227b83-d834-4276-bd61-27e1251a0708-workload-certs\") pod \"e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m\" (UID: \"a5227b83-d834-4276-bd61-27e1251a0708\") " pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" Oct 30 02:06:47.387244 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:47.387216 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5227b83-d834-4276-bd61-27e1251a0708-istiod-ca-cert\") pod \"e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m\" (UID: \"a5227b83-d834-4276-bd61-27e1251a0708\") " pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" Oct 30 02:06:47.387338 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:47.387304 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5227b83-d834-4276-bd61-27e1251a0708-workload-socket\") pod \"e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m\" (UID: \"a5227b83-d834-4276-bd61-27e1251a0708\") " pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" Oct 30 02:06:47.387753 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:47.387729 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5227b83-d834-4276-bd61-27e1251a0708-credential-socket\") pod \"e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m\" (UID: \"a5227b83-d834-4276-bd61-27e1251a0708\") " pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" Oct 30 02:06:47.415566 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:47.415349 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5227b83-d834-4276-bd61-27e1251a0708-istio-podinfo\") pod \"e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m\" (UID: \"a5227b83-d834-4276-bd61-27e1251a0708\") " pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" Oct 30 02:06:47.473019 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:47.435846 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5227b83-d834-4276-bd61-27e1251a0708-istio-envoy\") pod \"e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m\" (UID: \"a5227b83-d834-4276-bd61-27e1251a0708\") " pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" Oct 30 02:06:47.473019 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:47.460296 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5pq8\" (UniqueName: \"kubernetes.io/projected/a5227b83-d834-4276-bd61-27e1251a0708-kube-api-access-s5pq8\") pod \"e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m\" (UID: \"a5227b83-d834-4276-bd61-27e1251a0708\") " pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" Oct 30 02:06:47.475572 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:47.475499 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5227b83-d834-4276-bd61-27e1251a0708-istio-token\") pod \"e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m\" (UID: \"a5227b83-d834-4276-bd61-27e1251a0708\") " pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" Oct 30 02:06:47.559556 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:47.553255 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" Oct 30 02:06:47.882354 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:47.882321 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m"] Oct 30 02:06:47.884574 ip-10-0-135-65 kubenswrapper[6221]: W1030 02:06:47.884542 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5227b83_d834_4276_bd61_27e1251a0708.slice/crio-5d370e8c37b3fd8fe1e9254e24190df98959d9647cb425d8fcc4de683d1ebb88 WatchSource:0}: Error finding container 5d370e8c37b3fd8fe1e9254e24190df98959d9647cb425d8fcc4de683d1ebb88: Status 404 returned error can't find the container with id 5d370e8c37b3fd8fe1e9254e24190df98959d9647cb425d8fcc4de683d1ebb88 Oct 30 02:06:47.887595 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:47.887521 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:06:47.887704 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:47.887649 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:06:47.887704 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:47.887692 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:06:47.900306 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:47.900273 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" event={"ID":"a5227b83-d834-4276-bd61-27e1251a0708","Type":"ContainerStarted","Data":"5d370e8c37b3fd8fe1e9254e24190df98959d9647cb425d8fcc4de683d1ebb88"} Oct 30 02:06:48.386308 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:48.386272 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f89697-d7d8df5bb-2vnvq"] Oct 30 02:06:48.390945 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:48.390922 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f89697-d7d8df5bb-2vnvq" Oct 30 02:06:48.407778 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:48.407748 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-f89697\"" Oct 30 02:06:48.413457 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:48.413421 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f89697-d7d8df5bb-2vnvq"] Oct 30 02:06:48.503153 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:48.503098 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrt4f\" (UniqueName: \"kubernetes.io/projected/1153eb7c-0a5d-4bae-8894-1d1519393611-kube-api-access-zrt4f\") pod \"payload-processing-e2e-inf-a-f89697-d7d8df5bb-2vnvq\" (UID: \"1153eb7c-0a5d-4bae-8894-1d1519393611\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f89697-d7d8df5bb-2vnvq" Oct 30 02:06:48.503360 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:48.503292 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1153eb7c-0a5d-4bae-8894-1d1519393611-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f89697-d7d8df5bb-2vnvq\" (UID: \"1153eb7c-0a5d-4bae-8894-1d1519393611\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f89697-d7d8df5bb-2vnvq" Oct 30 02:06:48.668609 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:48.667021 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1153eb7c-0a5d-4bae-8894-1d1519393611-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f89697-d7d8df5bb-2vnvq\" (UID: \"1153eb7c-0a5d-4bae-8894-1d1519393611\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f89697-d7d8df5bb-2vnvq" Oct 30 02:06:48.668609 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:48.667297 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrt4f\" (UniqueName: \"kubernetes.io/projected/1153eb7c-0a5d-4bae-8894-1d1519393611-kube-api-access-zrt4f\") pod \"payload-processing-e2e-inf-a-f89697-d7d8df5bb-2vnvq\" (UID: \"1153eb7c-0a5d-4bae-8894-1d1519393611\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f89697-d7d8df5bb-2vnvq" Oct 30 02:06:48.673552 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:48.669061 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1153eb7c-0a5d-4bae-8894-1d1519393611-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f89697-d7d8df5bb-2vnvq\" (UID: \"1153eb7c-0a5d-4bae-8894-1d1519393611\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f89697-d7d8df5bb-2vnvq" Oct 30 02:06:48.710972 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:48.704362 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrt4f\" (UniqueName: \"kubernetes.io/projected/1153eb7c-0a5d-4bae-8894-1d1519393611-kube-api-access-zrt4f\") pod \"payload-processing-e2e-inf-a-f89697-d7d8df5bb-2vnvq\" (UID: \"1153eb7c-0a5d-4bae-8894-1d1519393611\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f89697-d7d8df5bb-2vnvq" Oct 30 02:06:48.723413 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:48.723378 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f89697-d7d8df5bb-2vnvq" Oct 30 02:06:48.909431 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:48.909386 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" event={"ID":"a5227b83-d834-4276-bd61-27e1251a0708","Type":"ContainerStarted","Data":"81581d75d759bd496f2d21225e362465f79d6d3968703b85be3cc502745b1332"} Oct 30 02:06:48.987633 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:48.976954 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" podStartSLOduration=1.976934903 podStartE2EDuration="1.976934903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:06:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:06:48.976920867 +0000 UTC m=+2932.555208204" watchObservedRunningTime="2026-10-30 02:06:48.976934903 +0000 UTC m=+2932.555222241" Oct 30 02:06:49.058457 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:49.058417 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f89697-d7d8df5bb-2vnvq"] Oct 30 02:06:49.062106 ip-10-0-135-65 kubenswrapper[6221]: W1030 02:06:49.062073 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1153eb7c_0a5d_4bae_8894_1d1519393611.slice/crio-2bf99f14f401bf803c2aadbc7c794e62756bd3d621dcc432ae55314a45e30ceb WatchSource:0}: Error finding container 2bf99f14f401bf803c2aadbc7c794e62756bd3d621dcc432ae55314a45e30ceb: Status 404 returned error can't find the container with id 2bf99f14f401bf803c2aadbc7c794e62756bd3d621dcc432ae55314a45e30ceb Oct 30 02:06:49.553908 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:49.553870 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" Oct 30 02:06:49.560968 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:49.560939 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" Oct 30 02:06:49.918589 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:49.918465 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f89697-d7d8df5bb-2vnvq" event={"ID":"1153eb7c-0a5d-4bae-8894-1d1519393611","Type":"ContainerStarted","Data":"073c3638974349ba741349a06b535863a4427e11e176c5e3e6c076c3cec34f28"} Oct 30 02:06:49.918589 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:49.918557 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f89697-d7d8df5bb-2vnvq" event={"ID":"1153eb7c-0a5d-4bae-8894-1d1519393611","Type":"ContainerStarted","Data":"2bf99f14f401bf803c2aadbc7c794e62756bd3d621dcc432ae55314a45e30ceb"} Oct 30 02:06:49.919070 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:49.918738 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" Oct 30 02:06:49.919070 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:49.918775 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f89697-d7d8df5bb-2vnvq" Oct 30 02:06:49.919727 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:49.919705 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" Oct 30 02:06:49.962919 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:49.962866 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-f89697-d7d8df5bb-2vnvq" podStartSLOduration=1.9628505760000001 podStartE2EDuration="1.962850576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:06:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:06:49.959476944 +0000 UTC m=+2933.537764280" watchObservedRunningTime="2026-10-30 02:06:49.962850576 +0000 UTC m=+2933.541137911" Oct 30 02:06:50.642987 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:50.642918 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh" Oct 30 02:06:50.827434 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:50.827391 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-workload-certs\") pod \"59eaf4dd-3059-4fa3-bac8-93fc2b4d518a\" (UID: \"59eaf4dd-3059-4fa3-bac8-93fc2b4d518a\") " Oct 30 02:06:50.827816 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:50.827777 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-istio-podinfo\") pod \"59eaf4dd-3059-4fa3-bac8-93fc2b4d518a\" (UID: \"59eaf4dd-3059-4fa3-bac8-93fc2b4d518a\") " Oct 30 02:06:50.827816 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:50.827797 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-workload-certs" pod "59eaf4dd-3059-4fa3-bac8-93fc2b4d518a" (UID: "59eaf4dd-3059-4fa3-bac8-93fc2b4d518a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:06:50.828044 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:50.827839 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-credential-socket\") pod \"59eaf4dd-3059-4fa3-bac8-93fc2b4d518a\" (UID: \"59eaf4dd-3059-4fa3-bac8-93fc2b4d518a\") " Oct 30 02:06:50.828044 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:50.827887 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-istiod-ca-cert\") pod \"59eaf4dd-3059-4fa3-bac8-93fc2b4d518a\" (UID: \"59eaf4dd-3059-4fa3-bac8-93fc2b4d518a\") " Oct 30 02:06:50.828044 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:50.827944 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-istio-envoy\") pod \"59eaf4dd-3059-4fa3-bac8-93fc2b4d518a\" (UID: \"59eaf4dd-3059-4fa3-bac8-93fc2b4d518a\") " Oct 30 02:06:50.828044 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:50.828011 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-istio-data\") pod \"59eaf4dd-3059-4fa3-bac8-93fc2b4d518a\" (UID: \"59eaf4dd-3059-4fa3-bac8-93fc2b4d518a\") " Oct 30 02:06:50.828282 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:50.828080 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-istio-token\" (UniqueName: \"kubernetes.io/projected/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-istio-token\") pod \"59eaf4dd-3059-4fa3-bac8-93fc2b4d518a\" (UID: \"59eaf4dd-3059-4fa3-bac8-93fc2b4d518a\") " Oct 30 02:06:50.828282 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:50.828169 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-kube-api-access-xzbhc\" (UniqueName: \"kubernetes.io/projected/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-kube-api-access-xzbhc\") pod \"59eaf4dd-3059-4fa3-bac8-93fc2b4d518a\" (UID: \"59eaf4dd-3059-4fa3-bac8-93fc2b4d518a\") " Oct 30 02:06:50.828399 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:50.828283 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-workload-socket\") pod \"59eaf4dd-3059-4fa3-bac8-93fc2b4d518a\" (UID: \"59eaf4dd-3059-4fa3-bac8-93fc2b4d518a\") " Oct 30 02:06:50.829050 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:50.828963 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:06:50.829348 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:50.829239 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-workload-socket" pod "59eaf4dd-3059-4fa3-bac8-93fc2b4d518a" (UID: "59eaf4dd-3059-4fa3-bac8-93fc2b4d518a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:06:50.838344 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:50.830066 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-istio-data" pod "59eaf4dd-3059-4fa3-bac8-93fc2b4d518a" (UID: "59eaf4dd-3059-4fa3-bac8-93fc2b4d518a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:06:50.838344 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:50.830263 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-credential-socket" pod "59eaf4dd-3059-4fa3-bac8-93fc2b4d518a" (UID: "59eaf4dd-3059-4fa3-bac8-93fc2b4d518a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:06:50.838344 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:50.830609 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-istiod-ca-cert" pod "59eaf4dd-3059-4fa3-bac8-93fc2b4d518a" (UID: "59eaf4dd-3059-4fa3-bac8-93fc2b4d518a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:06:50.840687 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:50.840634 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-istio-token" pod "59eaf4dd-3059-4fa3-bac8-93fc2b4d518a" (UID: "59eaf4dd-3059-4fa3-bac8-93fc2b4d518a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:50.840960 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:50.840934 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-kube-api-access-xzbhc" pod "59eaf4dd-3059-4fa3-bac8-93fc2b4d518a" (UID: "59eaf4dd-3059-4fa3-bac8-93fc2b4d518a"). InnerVolumeSpecName "kube-api-access-xzbhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:06:50.842232 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:50.842206 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-istio-envoy" pod "59eaf4dd-3059-4fa3-bac8-93fc2b4d518a" (UID: "59eaf4dd-3059-4fa3-bac8-93fc2b4d518a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:06:50.844009 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:50.843987 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-istio-podinfo" pod "59eaf4dd-3059-4fa3-bac8-93fc2b4d518a" (UID: "59eaf4dd-3059-4fa3-bac8-93fc2b4d518a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:06:50.937610 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:50.934870 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:06:50.937610 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:50.934908 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:06:50.937610 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:50.934923 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:06:50.937610 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:50.934937 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:06:50.937610 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:50.934949 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:06:50.937610 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:50.934961 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:06:50.937610 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:50.934974 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzbhc\" (UniqueName: \"kubernetes.io/projected/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-kube-api-access-xzbhc\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:06:50.937610 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:50.934988 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:06:50.958917 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:50.958879 6221 generic.go:356] "Generic (PLEG): container finished" podID="59eaf4dd-3059-4fa3-bac8-93fc2b4d518a" containerID="7ed4f067ebd5c37303404d57b1d652dd61a80ce5364d9e8b2907dc535a0f954d" exitCode=0 Oct 30 02:06:50.959103 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:50.958954 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh" Oct 30 02:06:50.959103 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:50.958980 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh" event={"ID":"59eaf4dd-3059-4fa3-bac8-93fc2b4d518a","Type":"ContainerDied","Data":"7ed4f067ebd5c37303404d57b1d652dd61a80ce5364d9e8b2907dc535a0f954d"} Oct 30 02:06:50.959103 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:50.959080 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh" event={"ID":"59eaf4dd-3059-4fa3-bac8-93fc2b4d518a","Type":"ContainerDied","Data":"34c7b2130b9c444193ed7a03ea256a0b3601de4bc56c2b2b2f418e5731b2209b"} Oct 30 02:06:50.959308 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:50.959112 6221 scope.go:122] "RemoveContainer" containerID="7ed4f067ebd5c37303404d57b1d652dd61a80ce5364d9e8b2907dc535a0f954d" Oct 30 02:06:50.972098 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:50.972078 6221 scope.go:122] "RemoveContainer" containerID="7ed4f067ebd5c37303404d57b1d652dd61a80ce5364d9e8b2907dc535a0f954d" Oct 30 02:06:50.972424 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:06:50.972406 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ed4f067ebd5c37303404d57b1d652dd61a80ce5364d9e8b2907dc535a0f954d\": container with ID starting with 7ed4f067ebd5c37303404d57b1d652dd61a80ce5364d9e8b2907dc535a0f954d not found: ID does not exist" containerID="7ed4f067ebd5c37303404d57b1d652dd61a80ce5364d9e8b2907dc535a0f954d" Oct 30 02:06:50.972476 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:50.972433 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ed4f067ebd5c37303404d57b1d652dd61a80ce5364d9e8b2907dc535a0f954d"} err="failed to get container status \"7ed4f067ebd5c37303404d57b1d652dd61a80ce5364d9e8b2907dc535a0f954d\": rpc error: code = NotFound desc = could not find container \"7ed4f067ebd5c37303404d57b1d652dd61a80ce5364d9e8b2907dc535a0f954d\": container with ID starting with 7ed4f067ebd5c37303404d57b1d652dd61a80ce5364d9e8b2907dc535a0f954d not found: ID does not exist" Oct 30 02:06:51.016466 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:51.016429 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh"] Oct 30 02:06:51.029828 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:51.029792 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-d55gh"] Oct 30 02:06:51.497195 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:06:51.497157 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 02:06:52.549774 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:52.549737 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59eaf4dd-3059-4fa3-bac8-93fc2b4d518a" path="/var/lib/kubelet/pods/59eaf4dd-3059-4fa3-bac8-93fc2b4d518a/volumes" Oct 30 02:06:53.277186 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:53.277151 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-aaf045-5f8df979f9-mmvp9"] Oct 30 02:06:53.277973 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:53.277956 6221 cpu_manager.go:414] "containerMap: removing container" podUID="59eaf4dd-3059-4fa3-bac8-93fc2b4d518a" containerName="istio-proxy" Oct 30 02:06:53.278037 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:53.277974 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59eaf4dd-3059-4fa3-bac8-93fc2b4d518a" containerName="istio-proxy" Oct 30 02:06:53.278161 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:53.278152 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="59eaf4dd-3059-4fa3-bac8-93fc2b4d518a" containerName="istio-proxy" Oct 30 02:06:53.283545 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:53.283512 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aaf045-5f8df979f9-mmvp9" Oct 30 02:06:53.307629 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:53.302438 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-aaf045-5f8df979f9-mmvp9"] Oct 30 02:06:53.310126 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:53.308569 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-aaf045-dockercfg-6rkpm\"" Oct 30 02:06:53.310126 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:53.308617 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-aaf045\"" Oct 30 02:06:53.498622 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:53.498585 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w77hn\" (UniqueName: \"kubernetes.io/projected/decbaf90-f664-408c-a7df-b7f2667f399b-kube-api-access-w77hn\") pod \"payload-pre-processing-e2e-inf-a-aaf045-5f8df979f9-mmvp9\" (UID: \"decbaf90-f664-408c-a7df-b7f2667f399b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aaf045-5f8df979f9-mmvp9" Oct 30 02:06:53.498814 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:53.498715 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/decbaf90-f664-408c-a7df-b7f2667f399b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-aaf045-5f8df979f9-mmvp9\" (UID: \"decbaf90-f664-408c-a7df-b7f2667f399b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aaf045-5f8df979f9-mmvp9" Oct 30 02:06:53.508473 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:53.508440 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-aaf045-64c758fc8b-jvf8m"] Oct 30 02:06:53.517403 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:53.517377 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-aaf045-64c758fc8b-jvf8m" Oct 30 02:06:53.520326 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:53.520296 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-aaf045-64c758fc8b-jvf8m"] Oct 30 02:06:53.600016 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:53.599974 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w77hn\" (UniqueName: \"kubernetes.io/projected/decbaf90-f664-408c-a7df-b7f2667f399b-kube-api-access-w77hn\") pod \"payload-pre-processing-e2e-inf-a-aaf045-5f8df979f9-mmvp9\" (UID: \"decbaf90-f664-408c-a7df-b7f2667f399b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aaf045-5f8df979f9-mmvp9" Oct 30 02:06:53.600505 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:53.600066 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq4fl\" (UniqueName: \"kubernetes.io/projected/c4ed9612-c80c-4768-a30d-5d43f2868e89-kube-api-access-rq4fl\") pod \"payload-processing-e2e-inf-a-aaf045-64c758fc8b-jvf8m\" (UID: \"c4ed9612-c80c-4768-a30d-5d43f2868e89\") " pod="openshift-ingress/payload-processing-e2e-inf-a-aaf045-64c758fc8b-jvf8m" Oct 30 02:06:53.600505 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:53.600109 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/decbaf90-f664-408c-a7df-b7f2667f399b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-aaf045-5f8df979f9-mmvp9\" (UID: \"decbaf90-f664-408c-a7df-b7f2667f399b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aaf045-5f8df979f9-mmvp9" Oct 30 02:06:53.600505 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:53.600351 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4ed9612-c80c-4768-a30d-5d43f2868e89-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-aaf045-64c758fc8b-jvf8m\" (UID: \"c4ed9612-c80c-4768-a30d-5d43f2868e89\") " pod="openshift-ingress/payload-processing-e2e-inf-a-aaf045-64c758fc8b-jvf8m" Oct 30 02:06:53.601017 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:53.600994 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/decbaf90-f664-408c-a7df-b7f2667f399b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-aaf045-5f8df979f9-mmvp9\" (UID: \"decbaf90-f664-408c-a7df-b7f2667f399b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aaf045-5f8df979f9-mmvp9" Oct 30 02:06:53.620707 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:53.620676 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w77hn\" (UniqueName: \"kubernetes.io/projected/decbaf90-f664-408c-a7df-b7f2667f399b-kube-api-access-w77hn\") pod \"payload-pre-processing-e2e-inf-a-aaf045-5f8df979f9-mmvp9\" (UID: \"decbaf90-f664-408c-a7df-b7f2667f399b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aaf045-5f8df979f9-mmvp9" Oct 30 02:06:53.629423 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:53.629402 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aaf045-5f8df979f9-mmvp9" Oct 30 02:06:53.705602 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:53.701406 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq4fl\" (UniqueName: \"kubernetes.io/projected/c4ed9612-c80c-4768-a30d-5d43f2868e89-kube-api-access-rq4fl\") pod \"payload-processing-e2e-inf-a-aaf045-64c758fc8b-jvf8m\" (UID: \"c4ed9612-c80c-4768-a30d-5d43f2868e89\") " pod="openshift-ingress/payload-processing-e2e-inf-a-aaf045-64c758fc8b-jvf8m" Oct 30 02:06:53.705602 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:53.701604 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4ed9612-c80c-4768-a30d-5d43f2868e89-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-aaf045-64c758fc8b-jvf8m\" (UID: \"c4ed9612-c80c-4768-a30d-5d43f2868e89\") " pod="openshift-ingress/payload-processing-e2e-inf-a-aaf045-64c758fc8b-jvf8m" Oct 30 02:06:53.705602 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:53.702630 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4ed9612-c80c-4768-a30d-5d43f2868e89-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-aaf045-64c758fc8b-jvf8m\" (UID: \"c4ed9612-c80c-4768-a30d-5d43f2868e89\") " pod="openshift-ingress/payload-processing-e2e-inf-a-aaf045-64c758fc8b-jvf8m" Oct 30 02:06:53.725601 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:53.725574 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq4fl\" (UniqueName: \"kubernetes.io/projected/c4ed9612-c80c-4768-a30d-5d43f2868e89-kube-api-access-rq4fl\") pod \"payload-processing-e2e-inf-a-aaf045-64c758fc8b-jvf8m\" (UID: \"c4ed9612-c80c-4768-a30d-5d43f2868e89\") " pod="openshift-ingress/payload-processing-e2e-inf-a-aaf045-64c758fc8b-jvf8m" Oct 30 02:06:53.836709 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:53.836675 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-aaf045-64c758fc8b-jvf8m" Oct 30 02:06:53.950142 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:53.949935 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-aaf045-5f8df979f9-mmvp9"] Oct 30 02:06:53.951295 ip-10-0-135-65 kubenswrapper[6221]: W1030 02:06:53.951258 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddecbaf90_f664_408c_a7df_b7f2667f399b.slice/crio-c39249ccb90530be985013a3bc07033686c2ba02e096bc94f0b4d29b4b4aee60 WatchSource:0}: Error finding container c39249ccb90530be985013a3bc07033686c2ba02e096bc94f0b4d29b4b4aee60: Status 404 returned error can't find the container with id c39249ccb90530be985013a3bc07033686c2ba02e096bc94f0b4d29b4b4aee60 Oct 30 02:06:53.985063 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:53.985034 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aaf045-5f8df979f9-mmvp9" event={"ID":"decbaf90-f664-408c-a7df-b7f2667f399b","Type":"ContainerStarted","Data":"c39249ccb90530be985013a3bc07033686c2ba02e096bc94f0b4d29b4b4aee60"} Oct 30 02:06:54.142570 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:54.142523 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-aaf045-64c758fc8b-jvf8m"] Oct 30 02:06:54.143741 ip-10-0-135-65 kubenswrapper[6221]: W1030 02:06:54.143715 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4ed9612_c80c_4768_a30d_5d43f2868e89.slice/crio-2d1507d518694418a1357799bf9f498a6aa167f96d7ecc359ae19ab11b85dbb5 WatchSource:0}: Error finding container 2d1507d518694418a1357799bf9f498a6aa167f96d7ecc359ae19ab11b85dbb5: Status 404 returned error can't find the container with id 2d1507d518694418a1357799bf9f498a6aa167f96d7ecc359ae19ab11b85dbb5 Oct 30 02:06:54.993214 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:54.993178 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-aaf045-64c758fc8b-jvf8m" event={"ID":"c4ed9612-c80c-4768-a30d-5d43f2868e89","Type":"ContainerStarted","Data":"309c549461f74a124d595f25e8ece6f9c436133ad7318cfc9479c21231ce2b0c"} Oct 30 02:06:54.993724 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:54.993244 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-aaf045-64c758fc8b-jvf8m" Oct 30 02:06:54.993724 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:54.993267 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-aaf045-64c758fc8b-jvf8m" event={"ID":"c4ed9612-c80c-4768-a30d-5d43f2868e89","Type":"ContainerStarted","Data":"2d1507d518694418a1357799bf9f498a6aa167f96d7ecc359ae19ab11b85dbb5"} Oct 30 02:06:54.994841 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:54.994818 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aaf045-5f8df979f9-mmvp9" event={"ID":"decbaf90-f664-408c-a7df-b7f2667f399b","Type":"ContainerStarted","Data":"a1bbbc5d25f508e0b330394313f3c087193f109f8321c004a794d241a5aac2bc"} Oct 30 02:06:54.994938 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:54.994865 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aaf045-5f8df979f9-mmvp9" Oct 30 02:06:55.037688 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:55.037636 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-aaf045-64c758fc8b-jvf8m" podStartSLOduration=2.037621856 podStartE2EDuration="2.037621856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:06:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:06:55.03325716 +0000 UTC m=+2938.611544495" watchObservedRunningTime="2026-10-30 02:06:55.037621856 +0000 UTC m=+2938.615909185" Oct 30 02:06:55.074065 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:55.074010 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aaf045-5f8df979f9-mmvp9" podStartSLOduration=2.073997335 podStartE2EDuration="2.073997335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:06:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:06:55.073190321 +0000 UTC m=+2938.651477657" watchObservedRunningTime="2026-10-30 02:06:55.073997335 +0000 UTC m=+2938.652284671" Oct 30 02:06:57.903944 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:06:57.903914 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-666655759c-gvdcm" Oct 30 02:07:00.963695 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:00.963664 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f89697-d7d8df5bb-2vnvq" Oct 30 02:07:06.005014 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:06.004977 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-aaf045-64c758fc8b-jvf8m" Oct 30 02:07:06.005604 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:06.005053 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aaf045-5f8df979f9-mmvp9" Oct 30 02:07:21.496958 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:07:21.496920 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 02:07:36.748987 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:36.748950 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 02:07:39.825118 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:39.825043 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-75bdfjd6z"] Oct 30 02:07:39.830884 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:39.830862 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-75bdfjd6z" Oct 30 02:07:39.866133 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:39.865902 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f1edaf\"/\"kube-root-ca.crt\"" Oct 30 02:07:39.866461 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:39.866433 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f1edaf\"/\"payload-processing-external-model-e2e--a558def5-dockercfg-z8l66\"" Oct 30 02:07:39.866597 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:39.866518 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-f1edaf\"/\"agc-cred-82ad3c391db1\"" Oct 30 02:07:39.866695 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:39.866670 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f1edaf\"/\"openshift-service-ca.crt\"" Oct 30 02:07:39.866755 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:39.866739 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f1edaf\"/\"payload-processing-external-model-plugins-e2e-extmod-b-f1edaf\"" Oct 30 02:07:39.866841 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:39.866826 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-f1edaf\"/\"routing-overlay\"" Oct 30 02:07:39.870270 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:39.870241 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/08205ca3-93a4-4353-971e-4614a6c9ccd8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1edaf-75bdfjd6z\" (UID: \"08205ca3-93a4-4353-971e-4614a6c9ccd8\") " pod="ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-75bdfjd6z" Oct 30 02:07:39.870450 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:39.870430 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08205ca3-93a4-4353-971e-4614a6c9ccd8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1edaf-75bdfjd6z\" (UID: \"08205ca3-93a4-4353-971e-4614a6c9ccd8\") " pod="ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-75bdfjd6z" Oct 30 02:07:39.870525 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:39.870484 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/08205ca3-93a4-4353-971e-4614a6c9ccd8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f1edaf-75bdfjd6z\" (UID: \"08205ca3-93a4-4353-971e-4614a6c9ccd8\") " pod="ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-75bdfjd6z" Oct 30 02:07:39.870650 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:39.870627 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfp48\" (UniqueName: \"kubernetes.io/projected/08205ca3-93a4-4353-971e-4614a6c9ccd8-kube-api-access-zfp48\") pod \"payload-processing-external-model-e2e-extmod-b-f1edaf-75bdfjd6z\" (UID: \"08205ca3-93a4-4353-971e-4614a6c9ccd8\") " pod="ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-75bdfjd6z" Oct 30 02:07:39.870693 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:39.870662 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-75bdfjd6z"] Oct 30 02:07:39.972763 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:39.972707 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08205ca3-93a4-4353-971e-4614a6c9ccd8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1edaf-75bdfjd6z\" (UID: \"08205ca3-93a4-4353-971e-4614a6c9ccd8\") " pod="ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-75bdfjd6z" Oct 30 02:07:39.973156 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:39.973130 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/08205ca3-93a4-4353-971e-4614a6c9ccd8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f1edaf-75bdfjd6z\" (UID: \"08205ca3-93a4-4353-971e-4614a6c9ccd8\") " pod="ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-75bdfjd6z" Oct 30 02:07:39.973412 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:39.973394 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfp48\" (UniqueName: \"kubernetes.io/projected/08205ca3-93a4-4353-971e-4614a6c9ccd8-kube-api-access-zfp48\") pod \"payload-processing-external-model-e2e-extmod-b-f1edaf-75bdfjd6z\" (UID: \"08205ca3-93a4-4353-971e-4614a6c9ccd8\") " pod="ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-75bdfjd6z" Oct 30 02:07:39.973801 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:39.973780 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/08205ca3-93a4-4353-971e-4614a6c9ccd8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1edaf-75bdfjd6z\" (UID: \"08205ca3-93a4-4353-971e-4614a6c9ccd8\") " pod="ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-75bdfjd6z" Oct 30 02:07:39.975884 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:39.974504 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08205ca3-93a4-4353-971e-4614a6c9ccd8-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1edaf-75bdfjd6z\" (UID: \"08205ca3-93a4-4353-971e-4614a6c9ccd8\") " pod="ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-75bdfjd6z" Oct 30 02:07:39.976056 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:39.974881 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/08205ca3-93a4-4353-971e-4614a6c9ccd8-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-f1edaf-75bdfjd6z\" (UID: \"08205ca3-93a4-4353-971e-4614a6c9ccd8\") " pod="ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-75bdfjd6z" Oct 30 02:07:40.000652 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:40.000611 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/08205ca3-93a4-4353-971e-4614a6c9ccd8-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-f1edaf-75bdfjd6z\" (UID: \"08205ca3-93a4-4353-971e-4614a6c9ccd8\") " pod="ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-75bdfjd6z" Oct 30 02:07:40.018083 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:40.017926 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfp48\" (UniqueName: \"kubernetes.io/projected/08205ca3-93a4-4353-971e-4614a6c9ccd8-kube-api-access-zfp48\") pod \"payload-processing-external-model-e2e-extmod-b-f1edaf-75bdfjd6z\" (UID: \"08205ca3-93a4-4353-971e-4614a6c9ccd8\") " pod="ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-75bdfjd6z" Oct 30 02:07:40.146345 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:40.146261 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-75bdfjd6z" Oct 30 02:07:40.498032 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:40.498003 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-75bdfjd6z"] Oct 30 02:07:40.500765 ip-10-0-135-65 kubenswrapper[6221]: W1030 02:07:40.500730 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08205ca3_93a4_4353_971e_4614a6c9ccd8.slice/crio-102092d0a184cbc3e3a09976abbc10f7e9a404d935d52fda077ddc1d47ea1b35 WatchSource:0}: Error finding container 102092d0a184cbc3e3a09976abbc10f7e9a404d935d52fda077ddc1d47ea1b35: Status 404 returned error can't find the container with id 102092d0a184cbc3e3a09976abbc10f7e9a404d935d52fda077ddc1d47ea1b35 Oct 30 02:07:41.514284 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:41.514246 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-75bdfjd6z" event={"ID":"08205ca3-93a4-4353-971e-4614a6c9ccd8","Type":"ContainerStarted","Data":"2715450e5fb302e13280f911d856c9666d96d173f41b28eca6c5e63f39ebfcc2"} Oct 30 02:07:41.514728 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:41.514311 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-75bdfjd6z" event={"ID":"08205ca3-93a4-4353-971e-4614a6c9ccd8","Type":"ContainerStarted","Data":"102092d0a184cbc3e3a09976abbc10f7e9a404d935d52fda077ddc1d47ea1b35"} Oct 30 02:07:41.514728 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:41.514341 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-75bdfjd6z" Oct 30 02:07:41.557937 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:41.557891 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-75bdfjd6z" podStartSLOduration=2.557876774 podStartE2EDuration="2.557876774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:07:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:07:41.555991805 +0000 UTC m=+2985.134279144" watchObservedRunningTime="2026-10-30 02:07:41.557876774 +0000 UTC m=+2985.136164127" Oct 30 02:07:47.202774 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:47.202738 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-666655759c-gvdcm"] Oct 30 02:07:51.566559 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:07:51.564235 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 02:07:52.203795 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:52.203750 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-666655759c-gvdcm" podUID="79f3f095-838b-4342-860b-0bfef6ffddae" containerName="payload-processing" containerID="cri-o://341c40c0e74c80b5e9fccd7bbd270d5be958fbd6d767876743a7600fe98b1482" gracePeriod=25 Oct 30 02:07:52.485342 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:52.485314 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-666655759c-gvdcm" Oct 30 02:07:52.517718 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:52.517691 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79f3f095-838b-4342-860b-0bfef6ffddae-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/79f3f095-838b-4342-860b-0bfef6ffddae-routing-overlay-volume\") pod \"79f3f095-838b-4342-860b-0bfef6ffddae\" (UID: \"79f3f095-838b-4342-860b-0bfef6ffddae\") " Oct 30 02:07:52.517868 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:52.517736 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79f3f095-838b-4342-860b-0bfef6ffddae-provider-credentials\" (UniqueName: \"kubernetes.io/projected/79f3f095-838b-4342-860b-0bfef6ffddae-provider-credentials\") pod \"79f3f095-838b-4342-860b-0bfef6ffddae\" (UID: \"79f3f095-838b-4342-860b-0bfef6ffddae\") " Oct 30 02:07:52.517976 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:52.517957 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79f3f095-838b-4342-860b-0bfef6ffddae-kube-api-access-gl68g\" (UniqueName: \"kubernetes.io/projected/79f3f095-838b-4342-860b-0bfef6ffddae-kube-api-access-gl68g\") pod \"79f3f095-838b-4342-860b-0bfef6ffddae\" (UID: \"79f3f095-838b-4342-860b-0bfef6ffddae\") " Oct 30 02:07:52.518088 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:52.518065 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79f3f095-838b-4342-860b-0bfef6ffddae-routing-overlay-volume" pod "79f3f095-838b-4342-860b-0bfef6ffddae" (UID: "79f3f095-838b-4342-860b-0bfef6ffddae"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:07:52.518160 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:52.518089 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79f3f095-838b-4342-860b-0bfef6ffddae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79f3f095-838b-4342-860b-0bfef6ffddae-plugins-config-volume\") pod \"79f3f095-838b-4342-860b-0bfef6ffddae\" (UID: \"79f3f095-838b-4342-860b-0bfef6ffddae\") " Oct 30 02:07:52.518455 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:52.518432 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79f3f095-838b-4342-860b-0bfef6ffddae-plugins-config-volume" pod "79f3f095-838b-4342-860b-0bfef6ffddae" (UID: "79f3f095-838b-4342-860b-0bfef6ffddae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:07:52.518764 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:52.518746 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79f3f095-838b-4342-860b-0bfef6ffddae-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:07:52.518843 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:52.518765 6221 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/79f3f095-838b-4342-860b-0bfef6ffddae-routing-overlay-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:07:52.520235 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:52.520208 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79f3f095-838b-4342-860b-0bfef6ffddae-kube-api-access-gl68g" pod "79f3f095-838b-4342-860b-0bfef6ffddae" (UID: "79f3f095-838b-4342-860b-0bfef6ffddae"). InnerVolumeSpecName "kube-api-access-gl68g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:07:52.520343 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:52.520258 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79f3f095-838b-4342-860b-0bfef6ffddae-provider-credentials" pod "79f3f095-838b-4342-860b-0bfef6ffddae" (UID: "79f3f095-838b-4342-860b-0bfef6ffddae"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:07:52.525125 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:52.525106 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-75bdfjd6z" Oct 30 02:07:52.603998 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:52.603967 6221 generic.go:356] "Generic (PLEG): container finished" podID="79f3f095-838b-4342-860b-0bfef6ffddae" containerID="341c40c0e74c80b5e9fccd7bbd270d5be958fbd6d767876743a7600fe98b1482" exitCode=0 Oct 30 02:07:52.604418 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:52.604010 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-666655759c-gvdcm" Oct 30 02:07:52.604418 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:52.604057 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-666655759c-gvdcm" event={"ID":"79f3f095-838b-4342-860b-0bfef6ffddae","Type":"ContainerDied","Data":"341c40c0e74c80b5e9fccd7bbd270d5be958fbd6d767876743a7600fe98b1482"} Oct 30 02:07:52.604418 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:52.604119 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-666655759c-gvdcm" event={"ID":"79f3f095-838b-4342-860b-0bfef6ffddae","Type":"ContainerDied","Data":"38e04ad49576532d2988067a3b645fb204cd13db0fa80084af97e920c27f520d"} Oct 30 02:07:52.604418 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:52.604146 6221 scope.go:122] "RemoveContainer" containerID="341c40c0e74c80b5e9fccd7bbd270d5be958fbd6d767876743a7600fe98b1482" Oct 30 02:07:52.615990 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:52.615970 6221 scope.go:122] "RemoveContainer" containerID="341c40c0e74c80b5e9fccd7bbd270d5be958fbd6d767876743a7600fe98b1482" Oct 30 02:07:52.616288 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:07:52.616264 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"341c40c0e74c80b5e9fccd7bbd270d5be958fbd6d767876743a7600fe98b1482\": container with ID starting with 341c40c0e74c80b5e9fccd7bbd270d5be958fbd6d767876743a7600fe98b1482 not found: ID does not exist" containerID="341c40c0e74c80b5e9fccd7bbd270d5be958fbd6d767876743a7600fe98b1482" Oct 30 02:07:52.616344 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:52.616298 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"341c40c0e74c80b5e9fccd7bbd270d5be958fbd6d767876743a7600fe98b1482"} err="failed to get container status \"341c40c0e74c80b5e9fccd7bbd270d5be958fbd6d767876743a7600fe98b1482\": rpc error: code = NotFound desc = could not find container \"341c40c0e74c80b5e9fccd7bbd270d5be958fbd6d767876743a7600fe98b1482\": container with ID starting with 341c40c0e74c80b5e9fccd7bbd270d5be958fbd6d767876743a7600fe98b1482 not found: ID does not exist" Oct 30 02:07:52.622695 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:52.622674 6221 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/79f3f095-838b-4342-860b-0bfef6ffddae-provider-credentials\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:07:52.622815 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:52.622699 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl68g\" (UniqueName: \"kubernetes.io/projected/79f3f095-838b-4342-860b-0bfef6ffddae-kube-api-access-gl68g\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:07:52.647571 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:52.647526 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-666655759c-gvdcm"] Oct 30 02:07:52.654735 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:52.654711 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-666655759c-gvdcm"] Oct 30 02:07:54.553335 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:54.553299 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79f3f095-838b-4342-860b-0bfef6ffddae" path="/var/lib/kubelet/pods/79f3f095-838b-4342-860b-0bfef6ffddae/volumes" Oct 30 02:07:57.126058 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:57.126028 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2vsns_d332bbb4-c699-44e8-96ba-516d8ca03e95/ovn-acl-logging/0.log" Oct 30 02:07:57.128849 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:07:57.128829 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2vsns_d332bbb4-c699-44e8-96ba-516d8ca03e95/ovn-acl-logging/0.log" Oct 30 02:08:05.347556 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:05.344370 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" podUID="ea9583aa-8259-4c9b-932e-2d6559008d86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 02:08:05.454222 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:05.454174 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" podUID="3b3c0e08-1371-4184-816a-d8b501526190" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:08:05.656728 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:05.656518 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" podUID="018c3fe2-3e67-45ed-a3d7-d70f3fe5b956" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 02:08:13.813086 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:13.813049 6221 generic.go:356] "Generic (PLEG): container finished" podID="a5227b83-d834-4276-bd61-27e1251a0708" containerID="81581d75d759bd496f2d21225e362465f79d6d3968703b85be3cc502745b1332" exitCode=0 Oct 30 02:08:13.813695 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:13.813120 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" event={"ID":"a5227b83-d834-4276-bd61-27e1251a0708","Type":"ContainerDied","Data":"81581d75d759bd496f2d21225e362465f79d6d3968703b85be3cc502745b1332"} Oct 30 02:08:13.813695 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:13.813572 6221 scope.go:122] "RemoveContainer" containerID="81581d75d759bd496f2d21225e362465f79d6d3968703b85be3cc502745b1332" Oct 30 02:08:13.815614 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:13.815582 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:08:13.815717 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:13.815655 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:08:13.815717 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:13.815684 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:08:14.836410 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:14.836372 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" event={"ID":"a5227b83-d834-4276-bd61-27e1251a0708","Type":"ContainerStarted","Data":"3a361174a51616ed91cd94aa605449f812dcc9e77a996937b6a1293f691464a0"} Oct 30 02:08:15.554284 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:15.554251 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" Oct 30 02:08:15.555835 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:15.555807 6221 patch_prober.go:29] interesting pod/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 30 02:08:15.555962 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:15.555854 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" podUID="a5227b83-d834-4276-bd61-27e1251a0708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 30 02:08:16.556301 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:16.556259 6221 patch_prober.go:29] interesting pod/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 30 02:08:16.556842 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:16.556329 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" podUID="a5227b83-d834-4276-bd61-27e1251a0708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 30 02:08:17.554958 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:17.554923 6221 patch_prober.go:29] interesting pod/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 30 02:08:17.555146 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:17.554982 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" podUID="a5227b83-d834-4276-bd61-27e1251a0708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 30 02:08:18.389971 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:18.389940 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-f81908/test-model-f81908-kserve-7c685775d7-szqcx"] Oct 30 02:08:18.391163 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:18.391137 6221 cpu_manager.go:414] "containerMap: removing container" podUID="79f3f095-838b-4342-860b-0bfef6ffddae" containerName="payload-processing" Oct 30 02:08:18.391163 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:18.391162 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79f3f095-838b-4342-860b-0bfef6ffddae" containerName="payload-processing" Oct 30 02:08:18.391381 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:18.391308 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="79f3f095-838b-4342-860b-0bfef6ffddae" containerName="payload-processing" Oct 30 02:08:18.394519 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:18.394495 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f81908/test-model-f81908-kserve-7c685775d7-szqcx" Oct 30 02:08:18.445947 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:18.445909 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f81908\"/\"kube-root-ca.crt\"" Oct 30 02:08:18.450828 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:18.450673 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f81908\"/\"test-model-f81908-kserve-self-signed-certs\"" Oct 30 02:08:18.450828 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:18.450734 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f81908\"/\"openshift-service-ca.crt\"" Oct 30 02:08:18.450828 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:18.450805 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f81908\"/\"default-dockercfg-wfcw9\"" Oct 30 02:08:18.455434 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:18.454609 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f81908/test-model-f81908-kserve-7c685775d7-szqcx"] Oct 30 02:08:18.546714 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:18.546682 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75b98edc-c54c-48c8-a6c2-b48f7d39f7be-tmp-dir\") pod \"test-model-f81908-kserve-7c685775d7-szqcx\" (UID: \"75b98edc-c54c-48c8-a6c2-b48f7d39f7be\") " pod="ai-tenant-e2e-inf-b-f81908/test-model-f81908-kserve-7c685775d7-szqcx" Oct 30 02:08:18.546908 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:18.546879 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh2dx\" (UniqueName: \"kubernetes.io/projected/75b98edc-c54c-48c8-a6c2-b48f7d39f7be-kube-api-access-jh2dx\") pod \"test-model-f81908-kserve-7c685775d7-szqcx\" (UID: \"75b98edc-c54c-48c8-a6c2-b48f7d39f7be\") " pod="ai-tenant-e2e-inf-b-f81908/test-model-f81908-kserve-7c685775d7-szqcx" Oct 30 02:08:18.547041 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:18.547021 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75b98edc-c54c-48c8-a6c2-b48f7d39f7be-home\") pod \"test-model-f81908-kserve-7c685775d7-szqcx\" (UID: \"75b98edc-c54c-48c8-a6c2-b48f7d39f7be\") " pod="ai-tenant-e2e-inf-b-f81908/test-model-f81908-kserve-7c685775d7-szqcx" Oct 30 02:08:18.547082 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:18.547049 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75b98edc-c54c-48c8-a6c2-b48f7d39f7be-tls-certs\") pod \"test-model-f81908-kserve-7c685775d7-szqcx\" (UID: \"75b98edc-c54c-48c8-a6c2-b48f7d39f7be\") " pod="ai-tenant-e2e-inf-b-f81908/test-model-f81908-kserve-7c685775d7-szqcx" Oct 30 02:08:18.547214 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:18.547196 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75b98edc-c54c-48c8-a6c2-b48f7d39f7be-dshm\") pod \"test-model-f81908-kserve-7c685775d7-szqcx\" (UID: \"75b98edc-c54c-48c8-a6c2-b48f7d39f7be\") " pod="ai-tenant-e2e-inf-b-f81908/test-model-f81908-kserve-7c685775d7-szqcx" Oct 30 02:08:18.547307 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:18.547292 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75b98edc-c54c-48c8-a6c2-b48f7d39f7be-model-cache\") pod \"test-model-f81908-kserve-7c685775d7-szqcx\" (UID: \"75b98edc-c54c-48c8-a6c2-b48f7d39f7be\") " pod="ai-tenant-e2e-inf-b-f81908/test-model-f81908-kserve-7c685775d7-szqcx" Oct 30 02:08:18.554173 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:18.554152 6221 patch_prober.go:29] interesting pod/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 30 02:08:18.554246 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:18.554196 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" podUID="a5227b83-d834-4276-bd61-27e1251a0708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 30 02:08:18.648467 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:18.648376 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75b98edc-c54c-48c8-a6c2-b48f7d39f7be-dshm\") pod \"test-model-f81908-kserve-7c685775d7-szqcx\" (UID: \"75b98edc-c54c-48c8-a6c2-b48f7d39f7be\") " pod="ai-tenant-e2e-inf-b-f81908/test-model-f81908-kserve-7c685775d7-szqcx" Oct 30 02:08:18.648467 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:18.648436 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75b98edc-c54c-48c8-a6c2-b48f7d39f7be-model-cache\") pod \"test-model-f81908-kserve-7c685775d7-szqcx\" (UID: \"75b98edc-c54c-48c8-a6c2-b48f7d39f7be\") " pod="ai-tenant-e2e-inf-b-f81908/test-model-f81908-kserve-7c685775d7-szqcx" Oct 30 02:08:18.648685 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:18.648497 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75b98edc-c54c-48c8-a6c2-b48f7d39f7be-tmp-dir\") pod \"test-model-f81908-kserve-7c685775d7-szqcx\" (UID: \"75b98edc-c54c-48c8-a6c2-b48f7d39f7be\") " pod="ai-tenant-e2e-inf-b-f81908/test-model-f81908-kserve-7c685775d7-szqcx" Oct 30 02:08:18.648685 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:18.648677 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh2dx\" (UniqueName: \"kubernetes.io/projected/75b98edc-c54c-48c8-a6c2-b48f7d39f7be-kube-api-access-jh2dx\") pod \"test-model-f81908-kserve-7c685775d7-szqcx\" (UID: \"75b98edc-c54c-48c8-a6c2-b48f7d39f7be\") " pod="ai-tenant-e2e-inf-b-f81908/test-model-f81908-kserve-7c685775d7-szqcx" Oct 30 02:08:18.648822 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:18.648806 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75b98edc-c54c-48c8-a6c2-b48f7d39f7be-home\") pod \"test-model-f81908-kserve-7c685775d7-szqcx\" (UID: \"75b98edc-c54c-48c8-a6c2-b48f7d39f7be\") " pod="ai-tenant-e2e-inf-b-f81908/test-model-f81908-kserve-7c685775d7-szqcx" Oct 30 02:08:18.648864 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:18.648836 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75b98edc-c54c-48c8-a6c2-b48f7d39f7be-tls-certs\") pod \"test-model-f81908-kserve-7c685775d7-szqcx\" (UID: \"75b98edc-c54c-48c8-a6c2-b48f7d39f7be\") " pod="ai-tenant-e2e-inf-b-f81908/test-model-f81908-kserve-7c685775d7-szqcx" Oct 30 02:08:18.648938 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:18.648913 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75b98edc-c54c-48c8-a6c2-b48f7d39f7be-model-cache\") pod \"test-model-f81908-kserve-7c685775d7-szqcx\" (UID: \"75b98edc-c54c-48c8-a6c2-b48f7d39f7be\") " pod="ai-tenant-e2e-inf-b-f81908/test-model-f81908-kserve-7c685775d7-szqcx" Oct 30 02:08:18.648986 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:18.648965 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75b98edc-c54c-48c8-a6c2-b48f7d39f7be-tmp-dir\") pod \"test-model-f81908-kserve-7c685775d7-szqcx\" (UID: \"75b98edc-c54c-48c8-a6c2-b48f7d39f7be\") " pod="ai-tenant-e2e-inf-b-f81908/test-model-f81908-kserve-7c685775d7-szqcx" Oct 30 02:08:18.649520 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:18.649355 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75b98edc-c54c-48c8-a6c2-b48f7d39f7be-home\") pod \"test-model-f81908-kserve-7c685775d7-szqcx\" (UID: \"75b98edc-c54c-48c8-a6c2-b48f7d39f7be\") " pod="ai-tenant-e2e-inf-b-f81908/test-model-f81908-kserve-7c685775d7-szqcx" Oct 30 02:08:18.651126 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:18.651097 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75b98edc-c54c-48c8-a6c2-b48f7d39f7be-dshm\") pod \"test-model-f81908-kserve-7c685775d7-szqcx\" (UID: \"75b98edc-c54c-48c8-a6c2-b48f7d39f7be\") " pod="ai-tenant-e2e-inf-b-f81908/test-model-f81908-kserve-7c685775d7-szqcx" Oct 30 02:08:18.651759 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:18.651738 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75b98edc-c54c-48c8-a6c2-b48f7d39f7be-tls-certs\") pod \"test-model-f81908-kserve-7c685775d7-szqcx\" (UID: \"75b98edc-c54c-48c8-a6c2-b48f7d39f7be\") " pod="ai-tenant-e2e-inf-b-f81908/test-model-f81908-kserve-7c685775d7-szqcx" Oct 30 02:08:18.668480 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:18.668457 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh2dx\" (UniqueName: \"kubernetes.io/projected/75b98edc-c54c-48c8-a6c2-b48f7d39f7be-kube-api-access-jh2dx\") pod \"test-model-f81908-kserve-7c685775d7-szqcx\" (UID: \"75b98edc-c54c-48c8-a6c2-b48f7d39f7be\") " pod="ai-tenant-e2e-inf-b-f81908/test-model-f81908-kserve-7c685775d7-szqcx" Oct 30 02:08:18.747563 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:18.746819 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f81908/test-model-f81908-kserve-7c685775d7-szqcx" Oct 30 02:08:19.194479 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:19.194453 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f81908/test-model-f81908-kserve-7c685775d7-szqcx"] Oct 30 02:08:19.196369 ip-10-0-135-65 kubenswrapper[6221]: W1030 02:08:19.196340 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75b98edc_c54c_48c8_a6c2_b48f7d39f7be.slice/crio-7cb6b58cc22706cf6fa4a840ad354ca874e150643be8bf6a511fec760930f32a WatchSource:0}: Error finding container 7cb6b58cc22706cf6fa4a840ad354ca874e150643be8bf6a511fec760930f32a: Status 404 returned error can't find the container with id 7cb6b58cc22706cf6fa4a840ad354ca874e150643be8bf6a511fec760930f32a Oct 30 02:08:19.554257 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:19.554226 6221 patch_prober.go:29] interesting pod/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" start-of-body= Oct 30 02:08:19.554700 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:19.554276 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" podUID="a5227b83-d834-4276-bd61-27e1251a0708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.201:15021/healthz/ready\": dial tcp 10.134.0.201:15021: connect: connection refused" Oct 30 02:08:19.895585 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:19.895458 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f81908/test-model-f81908-kserve-7c685775d7-szqcx" event={"ID":"75b98edc-c54c-48c8-a6c2-b48f7d39f7be","Type":"ContainerStarted","Data":"737ce7f70a023ef4b440bf54f8866bb6f0af82fe6e8bb1ed2a8c5c36bb7a8fb0"} Oct 30 02:08:19.895585 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:19.895499 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f81908/test-model-f81908-kserve-7c685775d7-szqcx" event={"ID":"75b98edc-c54c-48c8-a6c2-b48f7d39f7be","Type":"ContainerStarted","Data":"7cb6b58cc22706cf6fa4a840ad354ca874e150643be8bf6a511fec760930f32a"} Oct 30 02:08:19.928555 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:19.928503 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" Oct 30 02:08:19.950707 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:19.950654 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-f81908/test-model-f81908-kserve-7c685775d7-szqcx" podStartSLOduration=1.950637014 podStartE2EDuration="1.950637014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:08:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:08:19.950021541 +0000 UTC m=+3023.528308880" watchObservedRunningTime="2026-10-30 02:08:19.950637014 +0000 UTC m=+3023.528924351" Oct 30 02:08:20.558932 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:20.558902 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" Oct 30 02:08:20.748083 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:20.748054 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-f81908/test-model-f81908-kserve-7c685775d7-szqcx" Oct 30 02:08:20.903040 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:20.902958 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" Oct 30 02:08:21.495854 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:08:21.495819 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 02:08:28.748302 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:28.748264 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-f81908/test-model-f81908-kserve-7c685775d7-szqcx" Oct 30 02:08:28.761011 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:28.760984 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-f81908/test-model-f81908-kserve-7c685775d7-szqcx" Oct 30 02:08:28.969174 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:28.969144 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-f81908/test-model-f81908-kserve-7c685775d7-szqcx" Oct 30 02:08:34.680036 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:34.679983 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477"] Oct 30 02:08:34.714551 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:34.712845 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" Oct 30 02:08:34.762862 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:34.762822 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477"] Oct 30 02:08:34.777550 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:34.776067 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-22e99d-openshift-default-dockercfg-gvbsj\"" Oct 30 02:08:34.897492 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:34.897450 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0218485-8862-4117-902d-d5de5c8c963b-istio-podinfo\") pod \"e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477\" (UID: \"d0218485-8862-4117-902d-d5de5c8c963b\") " pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" Oct 30 02:08:34.897712 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:34.897501 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0218485-8862-4117-902d-d5de5c8c963b-istio-envoy\") pod \"e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477\" (UID: \"d0218485-8862-4117-902d-d5de5c8c963b\") " pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" Oct 30 02:08:34.897712 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:34.897688 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0218485-8862-4117-902d-d5de5c8c963b-credential-socket\") pod \"e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477\" (UID: \"d0218485-8862-4117-902d-d5de5c8c963b\") " pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" Oct 30 02:08:34.897823 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:34.897723 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0218485-8862-4117-902d-d5de5c8c963b-workload-certs\") pod \"e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477\" (UID: \"d0218485-8862-4117-902d-d5de5c8c963b\") " pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" Oct 30 02:08:34.897915 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:34.897883 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0218485-8862-4117-902d-d5de5c8c963b-istio-data\") pod \"e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477\" (UID: \"d0218485-8862-4117-902d-d5de5c8c963b\") " pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" Oct 30 02:08:34.898055 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:34.897938 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0218485-8862-4117-902d-d5de5c8c963b-istiod-ca-cert\") pod \"e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477\" (UID: \"d0218485-8862-4117-902d-d5de5c8c963b\") " pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" Oct 30 02:08:34.898111 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:34.898074 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z9vc\" (UniqueName: \"kubernetes.io/projected/d0218485-8862-4117-902d-d5de5c8c963b-kube-api-access-7z9vc\") pod \"e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477\" (UID: \"d0218485-8862-4117-902d-d5de5c8c963b\") " pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" Oct 30 02:08:34.898160 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:34.898114 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0218485-8862-4117-902d-d5de5c8c963b-istio-token\") pod \"e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477\" (UID: \"d0218485-8862-4117-902d-d5de5c8c963b\") " pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" Oct 30 02:08:34.898220 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:34.898207 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0218485-8862-4117-902d-d5de5c8c963b-workload-socket\") pod \"e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477\" (UID: \"d0218485-8862-4117-902d-d5de5c8c963b\") " pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" Oct 30 02:08:35.000182 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:35.000095 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0218485-8862-4117-902d-d5de5c8c963b-workload-socket\") pod \"e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477\" (UID: \"d0218485-8862-4117-902d-d5de5c8c963b\") " pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" Oct 30 02:08:35.000364 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:35.000187 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0218485-8862-4117-902d-d5de5c8c963b-istio-podinfo\") pod \"e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477\" (UID: \"d0218485-8862-4117-902d-d5de5c8c963b\") " pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" Oct 30 02:08:35.000364 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:35.000209 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0218485-8862-4117-902d-d5de5c8c963b-istio-envoy\") pod \"e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477\" (UID: \"d0218485-8862-4117-902d-d5de5c8c963b\") " pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" Oct 30 02:08:35.000364 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:35.000254 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0218485-8862-4117-902d-d5de5c8c963b-credential-socket\") pod \"e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477\" (UID: \"d0218485-8862-4117-902d-d5de5c8c963b\") " pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" Oct 30 02:08:35.000596 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:35.000370 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0218485-8862-4117-902d-d5de5c8c963b-workload-certs\") pod \"e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477\" (UID: \"d0218485-8862-4117-902d-d5de5c8c963b\") " pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" Oct 30 02:08:35.000596 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:35.000495 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0218485-8862-4117-902d-d5de5c8c963b-istio-data\") pod \"e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477\" (UID: \"d0218485-8862-4117-902d-d5de5c8c963b\") " pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" Oct 30 02:08:35.000596 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:35.000555 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0218485-8862-4117-902d-d5de5c8c963b-istiod-ca-cert\") pod \"e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477\" (UID: \"d0218485-8862-4117-902d-d5de5c8c963b\") " pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" Oct 30 02:08:35.000782 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:35.000608 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0218485-8862-4117-902d-d5de5c8c963b-workload-socket\") pod \"e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477\" (UID: \"d0218485-8862-4117-902d-d5de5c8c963b\") " pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" Oct 30 02:08:35.000782 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:35.000647 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0218485-8862-4117-902d-d5de5c8c963b-credential-socket\") pod \"e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477\" (UID: \"d0218485-8862-4117-902d-d5de5c8c963b\") " pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" Oct 30 02:08:35.000782 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:35.000652 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z9vc\" (UniqueName: \"kubernetes.io/projected/d0218485-8862-4117-902d-d5de5c8c963b-kube-api-access-7z9vc\") pod \"e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477\" (UID: \"d0218485-8862-4117-902d-d5de5c8c963b\") " pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" Oct 30 02:08:35.000782 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:35.000720 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0218485-8862-4117-902d-d5de5c8c963b-istio-token\") pod \"e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477\" (UID: \"d0218485-8862-4117-902d-d5de5c8c963b\") " pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" Oct 30 02:08:35.000782 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:35.000768 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0218485-8862-4117-902d-d5de5c8c963b-workload-certs\") pod \"e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477\" (UID: \"d0218485-8862-4117-902d-d5de5c8c963b\") " pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" Oct 30 02:08:35.001158 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:35.001133 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0218485-8862-4117-902d-d5de5c8c963b-istio-data\") pod \"e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477\" (UID: \"d0218485-8862-4117-902d-d5de5c8c963b\") " pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" Oct 30 02:08:35.001278 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:35.001259 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0218485-8862-4117-902d-d5de5c8c963b-istiod-ca-cert\") pod \"e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477\" (UID: \"d0218485-8862-4117-902d-d5de5c8c963b\") " pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" Oct 30 02:08:35.002656 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:35.002629 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0218485-8862-4117-902d-d5de5c8c963b-istio-envoy\") pod \"e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477\" (UID: \"d0218485-8862-4117-902d-d5de5c8c963b\") " pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" Oct 30 02:08:35.003013 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:35.002991 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0218485-8862-4117-902d-d5de5c8c963b-istio-podinfo\") pod \"e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477\" (UID: \"d0218485-8862-4117-902d-d5de5c8c963b\") " pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" Oct 30 02:08:35.021748 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:35.021721 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0218485-8862-4117-902d-d5de5c8c963b-istio-token\") pod \"e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477\" (UID: \"d0218485-8862-4117-902d-d5de5c8c963b\") " pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" Oct 30 02:08:35.021876 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:35.021795 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z9vc\" (UniqueName: \"kubernetes.io/projected/d0218485-8862-4117-902d-d5de5c8c963b-kube-api-access-7z9vc\") pod \"e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477\" (UID: \"d0218485-8862-4117-902d-d5de5c8c963b\") " pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" Oct 30 02:08:35.102032 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:35.101998 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" Oct 30 02:08:35.443143 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:35.443107 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477"] Oct 30 02:08:35.457127 ip-10-0-135-65 kubenswrapper[6221]: W1030 02:08:35.457090 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0218485_8862_4117_902d_d5de5c8c963b.slice/crio-3a53452d0c89200b67b6941bef2d245e4c72bf6fb4e8f5cdb8065cd035027091 WatchSource:0}: Error finding container 3a53452d0c89200b67b6941bef2d245e4c72bf6fb4e8f5cdb8065cd035027091: Status 404 returned error can't find the container with id 3a53452d0c89200b67b6941bef2d245e4c72bf6fb4e8f5cdb8065cd035027091 Oct 30 02:08:35.460938 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:35.460894 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:08:35.461048 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:35.460989 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:08:35.461048 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:35.461032 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:08:36.047268 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:36.047229 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" event={"ID":"d0218485-8862-4117-902d-d5de5c8c963b","Type":"ContainerStarted","Data":"c6e56cdfe0cf40702b3e4f5918a6a69450df03db19b2f4dac896ae545228e79d"} Oct 30 02:08:36.047833 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:36.047290 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" event={"ID":"d0218485-8862-4117-902d-d5de5c8c963b","Type":"ContainerStarted","Data":"3a53452d0c89200b67b6941bef2d245e4c72bf6fb4e8f5cdb8065cd035027091"} Oct 30 02:08:36.102692 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:36.102652 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" Oct 30 02:08:36.104794 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:36.104756 6221 patch_prober.go:29] interesting pod/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Oct 30 02:08:36.104967 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:36.104804 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" podUID="d0218485-8862-4117-902d-d5de5c8c963b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Oct 30 02:08:36.117391 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:36.117344 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" podStartSLOduration=2.117330795 podStartE2EDuration="2.117330795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:08:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:08:36.104223256 +0000 UTC m=+3039.682510594" watchObservedRunningTime="2026-10-30 02:08:36.117330795 +0000 UTC m=+3039.695618129" Oct 30 02:08:37.117970 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:37.117929 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" Oct 30 02:08:38.062605 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:38.062568 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" Oct 30 02:08:38.063508 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:38.063486 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" Oct 30 02:08:39.934640 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:39.934602 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 02:08:40.656588 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:40.625100 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-22e99d-7bc78986fb-wf66l"] Oct 30 02:08:40.703577 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:40.698818 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-22e99d-7bc78986fb-wf66l" Oct 30 02:08:40.719479 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:40.719446 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-22e99d-7bc78986fb-wf66l"] Oct 30 02:08:40.727598 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:40.727569 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-22e99d\"" Oct 30 02:08:40.727973 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:40.727952 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-22e99d-dockercfg-rdmdm\"" Oct 30 02:08:40.801938 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:40.801897 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj25w\" (UniqueName: \"kubernetes.io/projected/0c9898f9-d745-4345-97aa-7e82f95095c6-kube-api-access-sj25w\") pod \"payload-processing-e2e-inf-b-22e99d-7bc78986fb-wf66l\" (UID: \"0c9898f9-d745-4345-97aa-7e82f95095c6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-22e99d-7bc78986fb-wf66l" Oct 30 02:08:40.802131 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:40.802048 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c9898f9-d745-4345-97aa-7e82f95095c6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-22e99d-7bc78986fb-wf66l\" (UID: \"0c9898f9-d745-4345-97aa-7e82f95095c6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-22e99d-7bc78986fb-wf66l" Oct 30 02:08:40.903233 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:40.903192 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj25w\" (UniqueName: \"kubernetes.io/projected/0c9898f9-d745-4345-97aa-7e82f95095c6-kube-api-access-sj25w\") pod \"payload-processing-e2e-inf-b-22e99d-7bc78986fb-wf66l\" (UID: \"0c9898f9-d745-4345-97aa-7e82f95095c6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-22e99d-7bc78986fb-wf66l" Oct 30 02:08:40.903418 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:40.903321 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c9898f9-d745-4345-97aa-7e82f95095c6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-22e99d-7bc78986fb-wf66l\" (UID: \"0c9898f9-d745-4345-97aa-7e82f95095c6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-22e99d-7bc78986fb-wf66l" Oct 30 02:08:40.904145 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:40.904122 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c9898f9-d745-4345-97aa-7e82f95095c6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-22e99d-7bc78986fb-wf66l\" (UID: \"0c9898f9-d745-4345-97aa-7e82f95095c6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-22e99d-7bc78986fb-wf66l" Oct 30 02:08:40.929297 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:40.929225 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj25w\" (UniqueName: \"kubernetes.io/projected/0c9898f9-d745-4345-97aa-7e82f95095c6-kube-api-access-sj25w\") pod \"payload-processing-e2e-inf-b-22e99d-7bc78986fb-wf66l\" (UID: \"0c9898f9-d745-4345-97aa-7e82f95095c6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-22e99d-7bc78986fb-wf66l" Oct 30 02:08:41.065552 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:41.062059 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-22e99d-7bc78986fb-wf66l" Oct 30 02:08:41.497799 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:41.497772 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-22e99d-7bc78986fb-wf66l"] Oct 30 02:08:41.500625 ip-10-0-135-65 kubenswrapper[6221]: W1030 02:08:41.500592 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c9898f9_d745_4345_97aa_7e82f95095c6.slice/crio-5bd50cd1cd050491998c584b7c425d1130cfeeb456869335ac4abf7e4f3d732c WatchSource:0}: Error finding container 5bd50cd1cd050491998c584b7c425d1130cfeeb456869335ac4abf7e4f3d732c: Status 404 returned error can't find the container with id 5bd50cd1cd050491998c584b7c425d1130cfeeb456869335ac4abf7e4f3d732c Oct 30 02:08:42.126609 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:42.126572 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-22e99d-7bc78986fb-wf66l" event={"ID":"0c9898f9-d745-4345-97aa-7e82f95095c6","Type":"ContainerStarted","Data":"bdf8414db7276a0ee7958fff5e269580f44f229c37b2357ff6396eecd126b91e"} Oct 30 02:08:42.127373 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:42.126628 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-22e99d-7bc78986fb-wf66l" Oct 30 02:08:42.127373 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:42.126648 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-22e99d-7bc78986fb-wf66l" event={"ID":"0c9898f9-d745-4345-97aa-7e82f95095c6","Type":"ContainerStarted","Data":"5bd50cd1cd050491998c584b7c425d1130cfeeb456869335ac4abf7e4f3d732c"} Oct 30 02:08:42.170518 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:42.170463 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-22e99d-7bc78986fb-wf66l" podStartSLOduration=2.170447228 podStartE2EDuration="2.170447228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:08:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:08:42.167276606 +0000 UTC m=+3045.745563943" watchObservedRunningTime="2026-10-30 02:08:42.170447228 +0000 UTC m=+3045.748734560" Oct 30 02:08:51.497456 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:08:51.497415 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 02:08:53.137337 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:08:53.137301 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-22e99d-7bc78986fb-wf66l" Oct 30 02:09:06.605264 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:06.605224 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g"] Oct 30 02:09:06.615579 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:06.615512 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g" Oct 30 02:09:06.636013 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:06.635968 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g"] Oct 30 02:09:06.642833 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:06.642791 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-717065-openshift-default-dockercfg-6l8vj\"" Oct 30 02:09:06.698467 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:06.698427 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-credential-socket\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-62q4g\" (UID: \"4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g" Oct 30 02:09:06.698707 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:06.698583 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-istio-podinfo\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-62q4g\" (UID: \"4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g" Oct 30 02:09:06.698707 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:06.698619 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-istio-token\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-62q4g\" (UID: \"4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g" Oct 30 02:09:06.698707 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:06.698643 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc6n5\" (UniqueName: \"kubernetes.io/projected/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-kube-api-access-sc6n5\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-62q4g\" (UID: \"4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g" Oct 30 02:09:06.698887 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:06.698784 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-workload-certs\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-62q4g\" (UID: \"4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g" Oct 30 02:09:06.698887 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:06.698824 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-workload-socket\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-62q4g\" (UID: \"4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g" Oct 30 02:09:06.698887 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:06.698846 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-istio-envoy\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-62q4g\" (UID: \"4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g" Oct 30 02:09:06.699051 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:06.698885 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-istiod-ca-cert\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-62q4g\" (UID: \"4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g" Oct 30 02:09:06.699051 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:06.698987 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-istio-data\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-62q4g\" (UID: \"4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g" Oct 30 02:09:06.799959 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:06.799922 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-workload-socket\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-62q4g\" (UID: \"4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g" Oct 30 02:09:06.799959 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:06.799966 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-istio-envoy\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-62q4g\" (UID: \"4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g" Oct 30 02:09:06.800213 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:06.799988 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-istiod-ca-cert\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-62q4g\" (UID: \"4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g" Oct 30 02:09:06.800213 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:06.800045 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-istio-data\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-62q4g\" (UID: \"4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g" Oct 30 02:09:06.800213 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:06.800107 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-credential-socket\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-62q4g\" (UID: \"4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g" Oct 30 02:09:06.800393 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:06.800256 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-istio-podinfo\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-62q4g\" (UID: \"4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g" Oct 30 02:09:06.800393 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:06.800302 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-istio-token\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-62q4g\" (UID: \"4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g" Oct 30 02:09:06.800393 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:06.800339 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc6n5\" (UniqueName: \"kubernetes.io/projected/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-kube-api-access-sc6n5\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-62q4g\" (UID: \"4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g" Oct 30 02:09:06.800619 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:06.800420 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-workload-certs\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-62q4g\" (UID: \"4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g" Oct 30 02:09:06.800619 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:06.800460 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-workload-socket\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-62q4g\" (UID: \"4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g" Oct 30 02:09:06.800619 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:06.800591 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-istio-data\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-62q4g\" (UID: \"4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g" Oct 30 02:09:06.800788 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:06.800645 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-credential-socket\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-62q4g\" (UID: \"4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g" Oct 30 02:09:06.800847 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:06.800821 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-workload-certs\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-62q4g\" (UID: \"4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g" Oct 30 02:09:06.800919 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:06.800876 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-istiod-ca-cert\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-62q4g\" (UID: \"4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g" Oct 30 02:09:06.802885 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:06.802860 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-istio-envoy\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-62q4g\" (UID: \"4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g" Oct 30 02:09:06.803120 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:06.803101 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-istio-podinfo\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-62q4g\" (UID: \"4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g" Oct 30 02:09:06.821511 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:06.821469 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-istio-token\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-62q4g\" (UID: \"4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g" Oct 30 02:09:06.821941 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:06.821918 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc6n5\" (UniqueName: \"kubernetes.io/projected/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-kube-api-access-sc6n5\") pod \"e2e-inf-a-717065-openshift-default-6787fd448c-62q4g\" (UID: \"4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2\") " pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g" Oct 30 02:09:06.931696 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:06.931610 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g" Oct 30 02:09:07.253688 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:07.253661 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g"] Oct 30 02:09:07.256166 ip-10-0-135-65 kubenswrapper[6221]: W1030 02:09:07.256138 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b4b729b_a3fc_421d_bca1_3b7dcfd6b0a2.slice/crio-1280e83fbc02753515cf548d96c575be2e559268ffbdfb777a36936d2030c6e4 WatchSource:0}: Error finding container 1280e83fbc02753515cf548d96c575be2e559268ffbdfb777a36936d2030c6e4: Status 404 returned error can't find the container with id 1280e83fbc02753515cf548d96c575be2e559268ffbdfb777a36936d2030c6e4 Oct 30 02:09:07.259170 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:07.259134 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:09:07.259259 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:07.259226 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:09:07.259302 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:07.259264 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:09:07.347363 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:07.347329 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g" event={"ID":"4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2","Type":"ContainerStarted","Data":"464258c08cc04e9e13f1f8efdffe27e93eef437969e199be187a1128c43963f7"} Oct 30 02:09:07.347525 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:07.347382 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g" event={"ID":"4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2","Type":"ContainerStarted","Data":"1280e83fbc02753515cf548d96c575be2e559268ffbdfb777a36936d2030c6e4"} Oct 30 02:09:07.399338 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:07.399288 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g" podStartSLOduration=1.399271809 podStartE2EDuration="1.399271809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:09:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:09:07.397936574 +0000 UTC m=+3070.976223910" watchObservedRunningTime="2026-10-30 02:09:07.399271809 +0000 UTC m=+3070.977559145" Oct 30 02:09:07.932256 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:07.932223 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g" Oct 30 02:09:08.933880 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:08.933839 6221 patch_prober.go:29] interesting pod/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 30 02:09:08.934330 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:08.933915 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g" podUID="4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 30 02:09:09.933070 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:09.933030 6221 patch_prober.go:29] interesting pod/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 30 02:09:09.933396 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:09.933362 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g" podUID="4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 30 02:09:10.933051 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:10.933006 6221 patch_prober.go:29] interesting pod/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 30 02:09:10.933620 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:10.933078 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g" podUID="4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 30 02:09:11.932121 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:11.932084 6221 patch_prober.go:29] interesting pod/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 30 02:09:11.932121 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:11.932139 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g" podUID="4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 30 02:09:12.932072 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:12.932033 6221 patch_prober.go:29] interesting pod/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 30 02:09:12.932647 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:12.932096 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g" podUID="4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 30 02:09:13.932918 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:13.932875 6221 patch_prober.go:29] interesting pod/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 30 02:09:13.933388 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:13.932943 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g" podUID="4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 30 02:09:14.938236 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:14.938204 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g" Oct 30 02:09:14.938701 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:14.938626 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g" Oct 30 02:09:14.939478 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:14.939457 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g" Oct 30 02:09:21.496119 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:09:21.496080 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 02:09:43.212554 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:09:43.194593 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 02:09:51.496001 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:09:51.495965 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 02:10:00.349770 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:00.349735 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv"] Oct 30 02:10:00.354393 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:00.354371 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv" Oct 30 02:10:00.395786 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:00.395734 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c9548ee-90ef-4fe2-9412-4da31c20dfba-istio-podinfo\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv\" (UID: \"1c9548ee-90ef-4fe2-9412-4da31c20dfba\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv" Oct 30 02:10:00.395957 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:00.395792 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c9548ee-90ef-4fe2-9412-4da31c20dfba-istio-token\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv\" (UID: \"1c9548ee-90ef-4fe2-9412-4da31c20dfba\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv" Oct 30 02:10:00.396040 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:00.395998 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c9548ee-90ef-4fe2-9412-4da31c20dfba-workload-socket\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv\" (UID: \"1c9548ee-90ef-4fe2-9412-4da31c20dfba\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv" Oct 30 02:10:00.396154 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:00.396127 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c9548ee-90ef-4fe2-9412-4da31c20dfba-istiod-ca-cert\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv\" (UID: \"1c9548ee-90ef-4fe2-9412-4da31c20dfba\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv" Oct 30 02:10:00.396248 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:00.396214 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c9548ee-90ef-4fe2-9412-4da31c20dfba-istio-envoy\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv\" (UID: \"1c9548ee-90ef-4fe2-9412-4da31c20dfba\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv" Oct 30 02:10:00.396366 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:00.396346 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c9548ee-90ef-4fe2-9412-4da31c20dfba-credential-socket\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv\" (UID: \"1c9548ee-90ef-4fe2-9412-4da31c20dfba\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv" Oct 30 02:10:00.396465 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:00.396446 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c9548ee-90ef-4fe2-9412-4da31c20dfba-istio-data\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv\" (UID: \"1c9548ee-90ef-4fe2-9412-4da31c20dfba\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv" Oct 30 02:10:00.396560 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:00.396505 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnj9w\" (UniqueName: \"kubernetes.io/projected/1c9548ee-90ef-4fe2-9412-4da31c20dfba-kube-api-access-hnj9w\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv\" (UID: \"1c9548ee-90ef-4fe2-9412-4da31c20dfba\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv" Oct 30 02:10:00.396629 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:00.396567 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c9548ee-90ef-4fe2-9412-4da31c20dfba-workload-certs\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv\" (UID: \"1c9548ee-90ef-4fe2-9412-4da31c20dfba\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv" Oct 30 02:10:00.401422 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:00.401398 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-601d4b-openshift-default-dockercfg-7nw2n\"" Oct 30 02:10:00.407268 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:00.407240 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv"] Oct 30 02:10:00.497632 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:00.497601 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c9548ee-90ef-4fe2-9412-4da31c20dfba-istio-envoy\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv\" (UID: \"1c9548ee-90ef-4fe2-9412-4da31c20dfba\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv" Oct 30 02:10:00.497826 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:00.497669 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c9548ee-90ef-4fe2-9412-4da31c20dfba-credential-socket\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv\" (UID: \"1c9548ee-90ef-4fe2-9412-4da31c20dfba\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv" Oct 30 02:10:00.497826 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:00.497732 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c9548ee-90ef-4fe2-9412-4da31c20dfba-istio-data\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv\" (UID: \"1c9548ee-90ef-4fe2-9412-4da31c20dfba\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv" Oct 30 02:10:00.497826 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:00.497777 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnj9w\" (UniqueName: \"kubernetes.io/projected/1c9548ee-90ef-4fe2-9412-4da31c20dfba-kube-api-access-hnj9w\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv\" (UID: \"1c9548ee-90ef-4fe2-9412-4da31c20dfba\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv" Oct 30 02:10:00.497826 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:00.497803 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c9548ee-90ef-4fe2-9412-4da31c20dfba-workload-certs\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv\" (UID: \"1c9548ee-90ef-4fe2-9412-4da31c20dfba\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv" Oct 30 02:10:00.498055 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:00.497860 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c9548ee-90ef-4fe2-9412-4da31c20dfba-istio-podinfo\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv\" (UID: \"1c9548ee-90ef-4fe2-9412-4da31c20dfba\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv" Oct 30 02:10:00.498055 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:00.497892 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c9548ee-90ef-4fe2-9412-4da31c20dfba-istio-token\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv\" (UID: \"1c9548ee-90ef-4fe2-9412-4da31c20dfba\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv" Oct 30 02:10:00.498055 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:00.498027 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c9548ee-90ef-4fe2-9412-4da31c20dfba-workload-socket\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv\" (UID: \"1c9548ee-90ef-4fe2-9412-4da31c20dfba\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv" Oct 30 02:10:00.498222 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:00.498096 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c9548ee-90ef-4fe2-9412-4da31c20dfba-istiod-ca-cert\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv\" (UID: \"1c9548ee-90ef-4fe2-9412-4da31c20dfba\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv" Oct 30 02:10:00.498222 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:00.498145 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c9548ee-90ef-4fe2-9412-4da31c20dfba-credential-socket\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv\" (UID: \"1c9548ee-90ef-4fe2-9412-4da31c20dfba\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv" Oct 30 02:10:00.498341 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:00.498222 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c9548ee-90ef-4fe2-9412-4da31c20dfba-workload-certs\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv\" (UID: \"1c9548ee-90ef-4fe2-9412-4da31c20dfba\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv" Oct 30 02:10:00.498341 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:00.498261 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c9548ee-90ef-4fe2-9412-4da31c20dfba-istio-data\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv\" (UID: \"1c9548ee-90ef-4fe2-9412-4da31c20dfba\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv" Oct 30 02:10:00.498441 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:00.498428 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c9548ee-90ef-4fe2-9412-4da31c20dfba-workload-socket\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv\" (UID: \"1c9548ee-90ef-4fe2-9412-4da31c20dfba\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv" Oct 30 02:10:00.498977 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:00.498952 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c9548ee-90ef-4fe2-9412-4da31c20dfba-istiod-ca-cert\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv\" (UID: \"1c9548ee-90ef-4fe2-9412-4da31c20dfba\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv" Oct 30 02:10:00.500240 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:00.500208 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c9548ee-90ef-4fe2-9412-4da31c20dfba-istio-envoy\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv\" (UID: \"1c9548ee-90ef-4fe2-9412-4da31c20dfba\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv" Oct 30 02:10:00.500398 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:00.500381 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c9548ee-90ef-4fe2-9412-4da31c20dfba-istio-podinfo\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv\" (UID: \"1c9548ee-90ef-4fe2-9412-4da31c20dfba\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv" Oct 30 02:10:00.518463 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:00.518435 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c9548ee-90ef-4fe2-9412-4da31c20dfba-istio-token\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv\" (UID: \"1c9548ee-90ef-4fe2-9412-4da31c20dfba\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv" Oct 30 02:10:00.518647 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:00.518628 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnj9w\" (UniqueName: \"kubernetes.io/projected/1c9548ee-90ef-4fe2-9412-4da31c20dfba-kube-api-access-hnj9w\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv\" (UID: \"1c9548ee-90ef-4fe2-9412-4da31c20dfba\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv" Oct 30 02:10:00.669714 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:00.669625 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv" Oct 30 02:10:00.996685 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:00.996598 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv"] Oct 30 02:10:01.000197 ip-10-0-135-65 kubenswrapper[6221]: W1030 02:10:01.000164 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c9548ee_90ef_4fe2_9412_4da31c20dfba.slice/crio-e8a3dd0c455d1901edc3281760a37b2fd190b381469fa0a27765f58735efe955 WatchSource:0}: Error finding container e8a3dd0c455d1901edc3281760a37b2fd190b381469fa0a27765f58735efe955: Status 404 returned error can't find the container with id e8a3dd0c455d1901edc3281760a37b2fd190b381469fa0a27765f58735efe955 Oct 30 02:10:01.003240 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:01.003201 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:10:01.003361 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:01.003285 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:10:01.003361 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:01.003331 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:10:01.851581 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:01.851521 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv" event={"ID":"1c9548ee-90ef-4fe2-9412-4da31c20dfba","Type":"ContainerStarted","Data":"d828962be39b77681f25d73fdb539631b9c68dad4027b7262ee0cff69484f110"} Oct 30 02:10:01.852015 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:01.851608 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv" event={"ID":"1c9548ee-90ef-4fe2-9412-4da31c20dfba","Type":"ContainerStarted","Data":"e8a3dd0c455d1901edc3281760a37b2fd190b381469fa0a27765f58735efe955"} Oct 30 02:10:01.935565 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:01.935411 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv" podStartSLOduration=1.935388002 podStartE2EDuration="1.935388002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:10:01.927225908 +0000 UTC m=+3125.505513230" watchObservedRunningTime="2026-10-30 02:10:01.935388002 +0000 UTC m=+3125.513675342" Oct 30 02:10:02.670747 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:02.670706 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv" Oct 30 02:10:02.672412 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:02.672378 6221 patch_prober.go:29] interesting pod/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Oct 30 02:10:02.672597 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:02.672428 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv" podUID="1c9548ee-90ef-4fe2-9412-4da31c20dfba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Oct 30 02:10:03.670389 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:03.670354 6221 patch_prober.go:29] interesting pod/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Oct 30 02:10:03.670837 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:03.670410 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv" podUID="1c9548ee-90ef-4fe2-9412-4da31c20dfba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Oct 30 02:10:04.670177 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:04.670140 6221 patch_prober.go:29] interesting pod/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Oct 30 02:10:04.670365 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:04.670198 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv" podUID="1c9548ee-90ef-4fe2-9412-4da31c20dfba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Oct 30 02:10:05.671071 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:05.671025 6221 patch_prober.go:29] interesting pod/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Oct 30 02:10:05.671593 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:05.671088 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv" podUID="1c9548ee-90ef-4fe2-9412-4da31c20dfba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Oct 30 02:10:06.670733 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:06.670685 6221 patch_prober.go:29] interesting pod/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Oct 30 02:10:06.670940 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:06.670748 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv" podUID="1c9548ee-90ef-4fe2-9412-4da31c20dfba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Oct 30 02:10:07.675885 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:07.675851 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv" Oct 30 02:10:07.676394 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:07.676314 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv" Oct 30 02:10:07.677196 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:07.677167 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv" Oct 30 02:10:21.495028 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:10:21.494990 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 02:10:25.917144 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:25.917103 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 02:10:31.556431 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:31.556396 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-f89697/test-model-f89697-kserve-676c47d5c-ntkcd"] Oct 30 02:10:31.567889 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:31.567859 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f89697/test-model-f89697-kserve-676c47d5c-ntkcd" Oct 30 02:10:31.585840 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:31.585810 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f89697/test-model-f89697-kserve-676c47d5c-ntkcd"] Oct 30 02:10:31.592436 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:31.592403 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f89697\"/\"openshift-service-ca.crt\"" Oct 30 02:10:31.592584 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:31.592502 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f89697\"/\"kube-root-ca.crt\"" Oct 30 02:10:31.594942 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:31.594922 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f89697\"/\"test-model-f89697-kserve-self-signed-certs\"" Oct 30 02:10:31.594942 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:31.594926 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f89697\"/\"default-dockercfg-kjvbc\"" Oct 30 02:10:31.739320 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:31.739275 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1244d2b9-0346-460d-87bb-398ea191ee7b-home\") pod \"test-model-f89697-kserve-676c47d5c-ntkcd\" (UID: \"1244d2b9-0346-460d-87bb-398ea191ee7b\") " pod="ai-tenant-e2e-inf-a-f89697/test-model-f89697-kserve-676c47d5c-ntkcd" Oct 30 02:10:31.739556 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:31.739344 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1244d2b9-0346-460d-87bb-398ea191ee7b-model-cache\") pod \"test-model-f89697-kserve-676c47d5c-ntkcd\" (UID: \"1244d2b9-0346-460d-87bb-398ea191ee7b\") " pod="ai-tenant-e2e-inf-a-f89697/test-model-f89697-kserve-676c47d5c-ntkcd" Oct 30 02:10:31.739626 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:31.739567 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1244d2b9-0346-460d-87bb-398ea191ee7b-tls-certs\") pod \"test-model-f89697-kserve-676c47d5c-ntkcd\" (UID: \"1244d2b9-0346-460d-87bb-398ea191ee7b\") " pod="ai-tenant-e2e-inf-a-f89697/test-model-f89697-kserve-676c47d5c-ntkcd" Oct 30 02:10:31.739739 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:31.739723 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1244d2b9-0346-460d-87bb-398ea191ee7b-dshm\") pod \"test-model-f89697-kserve-676c47d5c-ntkcd\" (UID: \"1244d2b9-0346-460d-87bb-398ea191ee7b\") " pod="ai-tenant-e2e-inf-a-f89697/test-model-f89697-kserve-676c47d5c-ntkcd" Oct 30 02:10:31.739780 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:31.739747 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1244d2b9-0346-460d-87bb-398ea191ee7b-tmp-dir\") pod \"test-model-f89697-kserve-676c47d5c-ntkcd\" (UID: \"1244d2b9-0346-460d-87bb-398ea191ee7b\") " pod="ai-tenant-e2e-inf-a-f89697/test-model-f89697-kserve-676c47d5c-ntkcd" Oct 30 02:10:31.739780 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:31.739765 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smplv\" (UniqueName: \"kubernetes.io/projected/1244d2b9-0346-460d-87bb-398ea191ee7b-kube-api-access-smplv\") pod \"test-model-f89697-kserve-676c47d5c-ntkcd\" (UID: \"1244d2b9-0346-460d-87bb-398ea191ee7b\") " pod="ai-tenant-e2e-inf-a-f89697/test-model-f89697-kserve-676c47d5c-ntkcd" Oct 30 02:10:31.842649 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:31.842488 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1244d2b9-0346-460d-87bb-398ea191ee7b-tls-certs\") pod \"test-model-f89697-kserve-676c47d5c-ntkcd\" (UID: \"1244d2b9-0346-460d-87bb-398ea191ee7b\") " pod="ai-tenant-e2e-inf-a-f89697/test-model-f89697-kserve-676c47d5c-ntkcd" Oct 30 02:10:31.843167 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:31.843138 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1244d2b9-0346-460d-87bb-398ea191ee7b-dshm\") pod \"test-model-f89697-kserve-676c47d5c-ntkcd\" (UID: \"1244d2b9-0346-460d-87bb-398ea191ee7b\") " pod="ai-tenant-e2e-inf-a-f89697/test-model-f89697-kserve-676c47d5c-ntkcd" Oct 30 02:10:31.844406 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:31.844383 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1244d2b9-0346-460d-87bb-398ea191ee7b-tmp-dir\") pod \"test-model-f89697-kserve-676c47d5c-ntkcd\" (UID: \"1244d2b9-0346-460d-87bb-398ea191ee7b\") " pod="ai-tenant-e2e-inf-a-f89697/test-model-f89697-kserve-676c47d5c-ntkcd" Oct 30 02:10:31.852623 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:31.852592 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smplv\" (UniqueName: \"kubernetes.io/projected/1244d2b9-0346-460d-87bb-398ea191ee7b-kube-api-access-smplv\") pod \"test-model-f89697-kserve-676c47d5c-ntkcd\" (UID: \"1244d2b9-0346-460d-87bb-398ea191ee7b\") " pod="ai-tenant-e2e-inf-a-f89697/test-model-f89697-kserve-676c47d5c-ntkcd" Oct 30 02:10:31.854943 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:31.854918 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1244d2b9-0346-460d-87bb-398ea191ee7b-home\") pod \"test-model-f89697-kserve-676c47d5c-ntkcd\" (UID: \"1244d2b9-0346-460d-87bb-398ea191ee7b\") " pod="ai-tenant-e2e-inf-a-f89697/test-model-f89697-kserve-676c47d5c-ntkcd" Oct 30 02:10:31.855149 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:31.855134 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1244d2b9-0346-460d-87bb-398ea191ee7b-model-cache\") pod \"test-model-f89697-kserve-676c47d5c-ntkcd\" (UID: \"1244d2b9-0346-460d-87bb-398ea191ee7b\") " pod="ai-tenant-e2e-inf-a-f89697/test-model-f89697-kserve-676c47d5c-ntkcd" Oct 30 02:10:31.856612 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:31.855858 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1244d2b9-0346-460d-87bb-398ea191ee7b-tls-certs\") pod \"test-model-f89697-kserve-676c47d5c-ntkcd\" (UID: \"1244d2b9-0346-460d-87bb-398ea191ee7b\") " pod="ai-tenant-e2e-inf-a-f89697/test-model-f89697-kserve-676c47d5c-ntkcd" Oct 30 02:10:31.856997 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:31.848995 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1244d2b9-0346-460d-87bb-398ea191ee7b-tmp-dir\") pod \"test-model-f89697-kserve-676c47d5c-ntkcd\" (UID: \"1244d2b9-0346-460d-87bb-398ea191ee7b\") " pod="ai-tenant-e2e-inf-a-f89697/test-model-f89697-kserve-676c47d5c-ntkcd" Oct 30 02:10:31.857114 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:31.856095 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1244d2b9-0346-460d-87bb-398ea191ee7b-home\") pod \"test-model-f89697-kserve-676c47d5c-ntkcd\" (UID: \"1244d2b9-0346-460d-87bb-398ea191ee7b\") " pod="ai-tenant-e2e-inf-a-f89697/test-model-f89697-kserve-676c47d5c-ntkcd" Oct 30 02:10:31.857220 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:31.856301 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1244d2b9-0346-460d-87bb-398ea191ee7b-model-cache\") pod \"test-model-f89697-kserve-676c47d5c-ntkcd\" (UID: \"1244d2b9-0346-460d-87bb-398ea191ee7b\") " pod="ai-tenant-e2e-inf-a-f89697/test-model-f89697-kserve-676c47d5c-ntkcd" Oct 30 02:10:31.857321 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:31.856946 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1244d2b9-0346-460d-87bb-398ea191ee7b-dshm\") pod \"test-model-f89697-kserve-676c47d5c-ntkcd\" (UID: \"1244d2b9-0346-460d-87bb-398ea191ee7b\") " pod="ai-tenant-e2e-inf-a-f89697/test-model-f89697-kserve-676c47d5c-ntkcd" Oct 30 02:10:31.893947 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:31.893851 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smplv\" (UniqueName: \"kubernetes.io/projected/1244d2b9-0346-460d-87bb-398ea191ee7b-kube-api-access-smplv\") pod \"test-model-f89697-kserve-676c47d5c-ntkcd\" (UID: \"1244d2b9-0346-460d-87bb-398ea191ee7b\") " pod="ai-tenant-e2e-inf-a-f89697/test-model-f89697-kserve-676c47d5c-ntkcd" Oct 30 02:10:32.193742 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:32.193648 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f89697/test-model-f89697-kserve-676c47d5c-ntkcd" Oct 30 02:10:32.539670 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:32.539641 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f89697/test-model-f89697-kserve-676c47d5c-ntkcd"] Oct 30 02:10:32.541186 ip-10-0-135-65 kubenswrapper[6221]: W1030 02:10:32.541149 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1244d2b9_0346_460d_87bb_398ea191ee7b.slice/crio-652a2bdd63da07b012b3522cef771b94a21a9cb5e06dc6364e63a2d3361cf75f WatchSource:0}: Error finding container 652a2bdd63da07b012b3522cef771b94a21a9cb5e06dc6364e63a2d3361cf75f: Status 404 returned error can't find the container with id 652a2bdd63da07b012b3522cef771b94a21a9cb5e06dc6364e63a2d3361cf75f Oct 30 02:10:33.193561 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:33.193484 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f89697/test-model-f89697-kserve-676c47d5c-ntkcd" event={"ID":"1244d2b9-0346-460d-87bb-398ea191ee7b","Type":"ContainerStarted","Data":"2f07226e9586eaa15fc89858df37cfc00f96a61e2b13f97838d1d3ed95228b66"} Oct 30 02:10:33.193561 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:33.193555 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f89697/test-model-f89697-kserve-676c47d5c-ntkcd" event={"ID":"1244d2b9-0346-460d-87bb-398ea191ee7b","Type":"ContainerStarted","Data":"652a2bdd63da07b012b3522cef771b94a21a9cb5e06dc6364e63a2d3361cf75f"} Oct 30 02:10:33.194087 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:33.194074 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f89697/test-model-f89697-kserve-676c47d5c-ntkcd" Oct 30 02:10:33.254819 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:33.254760 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-f89697/test-model-f89697-kserve-676c47d5c-ntkcd" podStartSLOduration=2.254741234 podStartE2EDuration="2.254741234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:10:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:10:33.253539345 +0000 UTC m=+3156.831826673" watchObservedRunningTime="2026-10-30 02:10:33.254741234 +0000 UTC m=+3156.833028573" Oct 30 02:10:38.064817 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:38.064783 6221 patch_prober.go:29] interesting pod/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Oct 30 02:10:38.065313 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:38.064840 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" podUID="d0218485-8862-4117-902d-d5de5c8c963b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Oct 30 02:10:38.240468 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:38.240435 6221 generic.go:356] "Generic (PLEG): container finished" podID="d0218485-8862-4117-902d-d5de5c8c963b" containerID="c6e56cdfe0cf40702b3e4f5918a6a69450df03db19b2f4dac896ae545228e79d" exitCode=0 Oct 30 02:10:38.240705 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:38.240510 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" event={"ID":"d0218485-8862-4117-902d-d5de5c8c963b","Type":"ContainerDied","Data":"c6e56cdfe0cf40702b3e4f5918a6a69450df03db19b2f4dac896ae545228e79d"} Oct 30 02:10:38.241076 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:38.241057 6221 scope.go:122] "RemoveContainer" containerID="c6e56cdfe0cf40702b3e4f5918a6a69450df03db19b2f4dac896ae545228e79d" Oct 30 02:10:38.243125 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:38.243093 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:10:38.243237 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:38.243167 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:10:38.243237 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:38.243203 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:10:39.250624 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:39.250586 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" event={"ID":"d0218485-8862-4117-902d-d5de5c8c963b","Type":"ContainerStarted","Data":"928f5cff627b8d17ccd90cfd16bd06e03a1891d2f6f6cff47babb39627a0e8dc"} Oct 30 02:10:40.104308 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:40.102434 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" Oct 30 02:10:40.104308 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:40.102481 6221 patch_prober.go:29] interesting pod/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Oct 30 02:10:40.104308 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:40.102580 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" podUID="d0218485-8862-4117-902d-d5de5c8c963b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Oct 30 02:10:41.102703 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:41.102667 6221 patch_prober.go:29] interesting pod/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Oct 30 02:10:41.103134 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:41.102726 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" podUID="d0218485-8862-4117-902d-d5de5c8c963b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Oct 30 02:10:42.102730 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:42.102695 6221 patch_prober.go:29] interesting pod/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Oct 30 02:10:42.103156 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:42.102756 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" podUID="d0218485-8862-4117-902d-d5de5c8c963b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Oct 30 02:10:42.195100 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:42.195066 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-f89697/test-model-f89697-kserve-676c47d5c-ntkcd" Oct 30 02:10:42.234001 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:42.233970 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-f89697/test-model-f89697-kserve-676c47d5c-ntkcd" Oct 30 02:10:42.283553 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:42.283509 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f89697/test-model-f89697-kserve-676c47d5c-ntkcd" Oct 30 02:10:43.102660 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:43.102621 6221 patch_prober.go:29] interesting pod/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Oct 30 02:10:43.102845 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:43.102677 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" podUID="d0218485-8862-4117-902d-d5de5c8c963b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Oct 30 02:10:44.102715 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:44.102678 6221 patch_prober.go:29] interesting pod/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" start-of-body= Oct 30 02:10:44.102899 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:44.102737 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" podUID="d0218485-8862-4117-902d-d5de5c8c963b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.207:15021/healthz/ready\": dial tcp 10.134.0.207:15021: connect: connection refused" Oct 30 02:10:45.108223 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.108191 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" Oct 30 02:10:45.108673 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.108650 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" Oct 30 02:10:45.109312 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.109295 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" Oct 30 02:10:45.474320 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.474238 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4"] Oct 30 02:10:45.477821 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.477801 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4" Oct 30 02:10:45.505352 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.505321 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4"] Oct 30 02:10:45.522393 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.522357 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd"] Oct 30 02:10:45.527241 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.527219 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd" Oct 30 02:10:45.554752 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.554717 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd"] Oct 30 02:10:45.607924 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.607888 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61c55830-d642-4daa-8692-5f4f409b9a1a-istiod-ca-cert\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd\" (UID: \"61c55830-d642-4daa-8692-5f4f409b9a1a\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd" Oct 30 02:10:45.608131 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.607941 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61c55830-d642-4daa-8692-5f4f409b9a1a-istio-podinfo\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd\" (UID: \"61c55830-d642-4daa-8692-5f4f409b9a1a\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd" Oct 30 02:10:45.608131 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.608050 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c55830-d642-4daa-8692-5f4f409b9a1a-credential-socket\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd\" (UID: \"61c55830-d642-4daa-8692-5f4f409b9a1a\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd" Oct 30 02:10:45.608131 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.608095 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-workload-certs\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4\" (UID: \"de9bc101-0002-4af2-a1fa-ae9c6be2ff73\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4" Oct 30 02:10:45.608319 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.608142 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-istio-token\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4\" (UID: \"de9bc101-0002-4af2-a1fa-ae9c6be2ff73\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4" Oct 30 02:10:45.608319 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.608190 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61c55830-d642-4daa-8692-5f4f409b9a1a-istio-data\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd\" (UID: \"61c55830-d642-4daa-8692-5f4f409b9a1a\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd" Oct 30 02:10:45.608319 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.608243 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrnt8\" (UniqueName: \"kubernetes.io/projected/61c55830-d642-4daa-8692-5f4f409b9a1a-kube-api-access-vrnt8\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd\" (UID: \"61c55830-d642-4daa-8692-5f4f409b9a1a\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd" Oct 30 02:10:45.608319 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.608307 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-istio-podinfo\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4\" (UID: \"de9bc101-0002-4af2-a1fa-ae9c6be2ff73\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4" Oct 30 02:10:45.608513 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.608400 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61c55830-d642-4daa-8692-5f4f409b9a1a-istio-envoy\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd\" (UID: \"61c55830-d642-4daa-8692-5f4f409b9a1a\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd" Oct 30 02:10:45.608513 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.608487 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c55830-d642-4daa-8692-5f4f409b9a1a-workload-socket\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd\" (UID: \"61c55830-d642-4daa-8692-5f4f409b9a1a\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd" Oct 30 02:10:45.608632 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.608595 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-workload-socket\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4\" (UID: \"de9bc101-0002-4af2-a1fa-ae9c6be2ff73\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4" Oct 30 02:10:45.608678 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.608646 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-istiod-ca-cert\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4\" (UID: \"de9bc101-0002-4af2-a1fa-ae9c6be2ff73\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4" Oct 30 02:10:45.608773 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.608691 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdrsb\" (UniqueName: \"kubernetes.io/projected/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-kube-api-access-kdrsb\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4\" (UID: \"de9bc101-0002-4af2-a1fa-ae9c6be2ff73\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4" Oct 30 02:10:45.608773 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.608718 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61c55830-d642-4daa-8692-5f4f409b9a1a-workload-certs\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd\" (UID: \"61c55830-d642-4daa-8692-5f4f409b9a1a\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd" Oct 30 02:10:45.608889 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.608807 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-istio-envoy\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4\" (UID: \"de9bc101-0002-4af2-a1fa-ae9c6be2ff73\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4" Oct 30 02:10:45.608889 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.608867 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-istio-data\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4\" (UID: \"de9bc101-0002-4af2-a1fa-ae9c6be2ff73\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4" Oct 30 02:10:45.609012 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.608906 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-credential-socket\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4\" (UID: \"de9bc101-0002-4af2-a1fa-ae9c6be2ff73\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4" Oct 30 02:10:45.609012 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.608934 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61c55830-d642-4daa-8692-5f4f409b9a1a-istio-token\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd\" (UID: \"61c55830-d642-4daa-8692-5f4f409b9a1a\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd" Oct 30 02:10:45.710521 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.710478 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61c55830-d642-4daa-8692-5f4f409b9a1a-istio-envoy\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd\" (UID: \"61c55830-d642-4daa-8692-5f4f409b9a1a\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd" Oct 30 02:10:45.710772 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.710614 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c55830-d642-4daa-8692-5f4f409b9a1a-workload-socket\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd\" (UID: \"61c55830-d642-4daa-8692-5f4f409b9a1a\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd" Oct 30 02:10:45.710772 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.710706 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-workload-socket\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4\" (UID: \"de9bc101-0002-4af2-a1fa-ae9c6be2ff73\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4" Oct 30 02:10:45.710772 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.710759 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-istiod-ca-cert\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4\" (UID: \"de9bc101-0002-4af2-a1fa-ae9c6be2ff73\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4" Oct 30 02:10:45.710950 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.710795 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdrsb\" (UniqueName: \"kubernetes.io/projected/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-kube-api-access-kdrsb\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4\" (UID: \"de9bc101-0002-4af2-a1fa-ae9c6be2ff73\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4" Oct 30 02:10:45.710950 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.710841 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61c55830-d642-4daa-8692-5f4f409b9a1a-workload-certs\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd\" (UID: \"61c55830-d642-4daa-8692-5f4f409b9a1a\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd" Oct 30 02:10:45.710950 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.710883 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-istio-envoy\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4\" (UID: \"de9bc101-0002-4af2-a1fa-ae9c6be2ff73\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4" Oct 30 02:10:45.710950 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.710935 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-istio-data\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4\" (UID: \"de9bc101-0002-4af2-a1fa-ae9c6be2ff73\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4" Oct 30 02:10:45.711168 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.710976 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-credential-socket\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4\" (UID: \"de9bc101-0002-4af2-a1fa-ae9c6be2ff73\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4" Oct 30 02:10:45.711168 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.711002 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61c55830-d642-4daa-8692-5f4f409b9a1a-istio-token\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd\" (UID: \"61c55830-d642-4daa-8692-5f4f409b9a1a\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd" Oct 30 02:10:45.711168 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.711055 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61c55830-d642-4daa-8692-5f4f409b9a1a-istiod-ca-cert\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd\" (UID: \"61c55830-d642-4daa-8692-5f4f409b9a1a\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd" Oct 30 02:10:45.711334 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.711175 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61c55830-d642-4daa-8692-5f4f409b9a1a-workload-certs\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd\" (UID: \"61c55830-d642-4daa-8692-5f4f409b9a1a\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd" Oct 30 02:10:45.711334 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.711185 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-workload-socket\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4\" (UID: \"de9bc101-0002-4af2-a1fa-ae9c6be2ff73\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4" Oct 30 02:10:45.711453 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.711375 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-credential-socket\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4\" (UID: \"de9bc101-0002-4af2-a1fa-ae9c6be2ff73\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4" Oct 30 02:10:45.711453 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.711421 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61c55830-d642-4daa-8692-5f4f409b9a1a-istio-podinfo\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd\" (UID: \"61c55830-d642-4daa-8692-5f4f409b9a1a\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd" Oct 30 02:10:45.711453 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.711447 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-istio-data\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4\" (UID: \"de9bc101-0002-4af2-a1fa-ae9c6be2ff73\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4" Oct 30 02:10:45.711663 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.711495 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c55830-d642-4daa-8692-5f4f409b9a1a-credential-socket\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd\" (UID: \"61c55830-d642-4daa-8692-5f4f409b9a1a\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd" Oct 30 02:10:45.711663 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.711564 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-workload-certs\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4\" (UID: \"de9bc101-0002-4af2-a1fa-ae9c6be2ff73\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4" Oct 30 02:10:45.711663 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.711590 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-istio-token\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4\" (UID: \"de9bc101-0002-4af2-a1fa-ae9c6be2ff73\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4" Oct 30 02:10:45.711663 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.711595 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c55830-d642-4daa-8692-5f4f409b9a1a-workload-socket\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd\" (UID: \"61c55830-d642-4daa-8692-5f4f409b9a1a\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd" Oct 30 02:10:45.711663 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.711614 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-istiod-ca-cert\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4\" (UID: \"de9bc101-0002-4af2-a1fa-ae9c6be2ff73\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4" Oct 30 02:10:45.711945 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.711683 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61c55830-d642-4daa-8692-5f4f409b9a1a-istio-data\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd\" (UID: \"61c55830-d642-4daa-8692-5f4f409b9a1a\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd" Oct 30 02:10:45.711945 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.711711 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrnt8\" (UniqueName: \"kubernetes.io/projected/61c55830-d642-4daa-8692-5f4f409b9a1a-kube-api-access-vrnt8\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd\" (UID: \"61c55830-d642-4daa-8692-5f4f409b9a1a\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd" Oct 30 02:10:45.711945 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.711786 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-istio-podinfo\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4\" (UID: \"de9bc101-0002-4af2-a1fa-ae9c6be2ff73\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4" Oct 30 02:10:45.711945 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.711822 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61c55830-d642-4daa-8692-5f4f409b9a1a-istiod-ca-cert\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd\" (UID: \"61c55830-d642-4daa-8692-5f4f409b9a1a\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd" Oct 30 02:10:45.712164 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.712091 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-workload-certs\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4\" (UID: \"de9bc101-0002-4af2-a1fa-ae9c6be2ff73\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4" Oct 30 02:10:45.712228 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.712194 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c55830-d642-4daa-8692-5f4f409b9a1a-credential-socket\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd\" (UID: \"61c55830-d642-4daa-8692-5f4f409b9a1a\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd" Oct 30 02:10:45.712333 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.712311 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61c55830-d642-4daa-8692-5f4f409b9a1a-istio-data\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd\" (UID: \"61c55830-d642-4daa-8692-5f4f409b9a1a\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd" Oct 30 02:10:45.713618 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.713592 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-istio-envoy\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4\" (UID: \"de9bc101-0002-4af2-a1fa-ae9c6be2ff73\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4" Oct 30 02:10:45.713618 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.713605 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61c55830-d642-4daa-8692-5f4f409b9a1a-istio-envoy\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd\" (UID: \"61c55830-d642-4daa-8692-5f4f409b9a1a\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd" Oct 30 02:10:45.714006 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.713983 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61c55830-d642-4daa-8692-5f4f409b9a1a-istio-podinfo\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd\" (UID: \"61c55830-d642-4daa-8692-5f4f409b9a1a\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd" Oct 30 02:10:45.714338 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.714316 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-istio-podinfo\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4\" (UID: \"de9bc101-0002-4af2-a1fa-ae9c6be2ff73\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4" Oct 30 02:10:45.731552 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.731471 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61c55830-d642-4daa-8692-5f4f409b9a1a-istio-token\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd\" (UID: \"61c55830-d642-4daa-8692-5f4f409b9a1a\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd" Oct 30 02:10:45.732054 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.732032 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrnt8\" (UniqueName: \"kubernetes.io/projected/61c55830-d642-4daa-8692-5f4f409b9a1a-kube-api-access-vrnt8\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd\" (UID: \"61c55830-d642-4daa-8692-5f4f409b9a1a\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd" Oct 30 02:10:45.732192 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.732174 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdrsb\" (UniqueName: \"kubernetes.io/projected/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-kube-api-access-kdrsb\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4\" (UID: \"de9bc101-0002-4af2-a1fa-ae9c6be2ff73\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4" Oct 30 02:10:45.732356 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.732338 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-istio-token\") pod \"e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4\" (UID: \"de9bc101-0002-4af2-a1fa-ae9c6be2ff73\") " pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4" Oct 30 02:10:45.791524 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.791489 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4" Oct 30 02:10:45.853425 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:45.853390 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd" Oct 30 02:10:46.124595 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:46.124560 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4"] Oct 30 02:10:46.128778 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:46.128738 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:10:46.128916 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:46.128870 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:10:46.128989 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:46.128963 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:10:46.186958 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:46.186928 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd"] Oct 30 02:10:46.187988 ip-10-0-135-65 kubenswrapper[6221]: W1030 02:10:46.187965 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61c55830_d642_4daa_8692_5f4f409b9a1a.slice/crio-c88c7a93ca9df4a54f9319fd2d2ec383530a012954a8892add44d58dc42d1555 WatchSource:0}: Error finding container c88c7a93ca9df4a54f9319fd2d2ec383530a012954a8892add44d58dc42d1555: Status 404 returned error can't find the container with id c88c7a93ca9df4a54f9319fd2d2ec383530a012954a8892add44d58dc42d1555 Oct 30 02:10:46.191210 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:46.191177 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:10:46.191292 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:46.191264 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:10:46.191337 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:46.191293 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:10:46.307176 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:46.306824 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd" event={"ID":"61c55830-d642-4daa-8692-5f4f409b9a1a","Type":"ContainerStarted","Data":"517da38a773a77e45280517c2ed45aaf470a5296a9e26718e796ab3dc4858c42"} Oct 30 02:10:46.307176 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:46.306895 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd" event={"ID":"61c55830-d642-4daa-8692-5f4f409b9a1a","Type":"ContainerStarted","Data":"c88c7a93ca9df4a54f9319fd2d2ec383530a012954a8892add44d58dc42d1555"} Oct 30 02:10:46.309472 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:46.309440 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4" event={"ID":"de9bc101-0002-4af2-a1fa-ae9c6be2ff73","Type":"ContainerStarted","Data":"7750b5c6e89c2dce8a4b1576b879fcdc33affd79f32bb3879266be3ac216d8d3"} Oct 30 02:10:46.309647 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:46.309496 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4" event={"ID":"de9bc101-0002-4af2-a1fa-ae9c6be2ff73","Type":"ContainerStarted","Data":"b643c12945f451573d713e0c65e15db94a9bdea03585adfa74e2cc91e29a3fc6"} Oct 30 02:10:46.359573 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:46.357559 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd" podStartSLOduration=1.357525314 podStartE2EDuration="1.357525314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:10:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:10:46.354704874 +0000 UTC m=+3169.932992232" watchObservedRunningTime="2026-10-30 02:10:46.357525314 +0000 UTC m=+3169.935812651" Oct 30 02:10:46.401551 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:46.401416 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4" podStartSLOduration=1.401396114 podStartE2EDuration="1.401396114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:10:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:10:46.400023934 +0000 UTC m=+3169.978311271" watchObservedRunningTime="2026-10-30 02:10:46.401396114 +0000 UTC m=+3169.979683452" Oct 30 02:10:46.792242 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:46.792210 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4" Oct 30 02:10:46.853849 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:46.853816 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd" Oct 30 02:10:47.794308 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:47.794268 6221 patch_prober.go:29] interesting pod/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Oct 30 02:10:47.794732 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:47.794319 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4" podUID="de9bc101-0002-4af2-a1fa-ae9c6be2ff73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Oct 30 02:10:47.860816 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:47.860766 6221 patch_prober.go:29] interesting pod/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Oct 30 02:10:47.861031 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:47.860833 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd" podUID="61c55830-d642-4daa-8692-5f4f409b9a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Oct 30 02:10:48.791941 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:48.791907 6221 patch_prober.go:29] interesting pod/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Oct 30 02:10:48.792116 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:48.791958 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4" podUID="de9bc101-0002-4af2-a1fa-ae9c6be2ff73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Oct 30 02:10:48.855024 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:48.854984 6221 patch_prober.go:29] interesting pod/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Oct 30 02:10:48.855701 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:48.855654 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd" podUID="61c55830-d642-4daa-8692-5f4f409b9a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Oct 30 02:10:49.792818 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:49.792782 6221 patch_prober.go:29] interesting pod/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Oct 30 02:10:49.792993 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:49.792832 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4" podUID="de9bc101-0002-4af2-a1fa-ae9c6be2ff73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Oct 30 02:10:49.855295 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:49.855258 6221 patch_prober.go:29] interesting pod/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Oct 30 02:10:49.855993 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:49.855942 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd" podUID="61c55830-d642-4daa-8692-5f4f409b9a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Oct 30 02:10:50.791810 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:50.791766 6221 patch_prober.go:29] interesting pod/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Oct 30 02:10:50.792037 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:50.791816 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4" podUID="de9bc101-0002-4af2-a1fa-ae9c6be2ff73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Oct 30 02:10:50.855896 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:50.855858 6221 patch_prober.go:29] interesting pod/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Oct 30 02:10:50.856505 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:50.856473 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd" podUID="61c55830-d642-4daa-8692-5f4f409b9a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Oct 30 02:10:51.501590 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:10:51.501558 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 02:10:51.792334 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:51.792304 6221 patch_prober.go:29] interesting pod/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Oct 30 02:10:51.792548 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:51.792359 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4" podUID="de9bc101-0002-4af2-a1fa-ae9c6be2ff73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Oct 30 02:10:51.855250 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:51.855209 6221 patch_prober.go:29] interesting pod/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Oct 30 02:10:51.856280 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:51.856247 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd" podUID="61c55830-d642-4daa-8692-5f4f409b9a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Oct 30 02:10:52.792809 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:52.792770 6221 patch_prober.go:29] interesting pod/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" start-of-body= Oct 30 02:10:52.793039 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:52.792825 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4" podUID="de9bc101-0002-4af2-a1fa-ae9c6be2ff73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.212:15021/healthz/ready\": dial tcp 10.134.0.212:15021: connect: connection refused" Oct 30 02:10:52.853884 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:52.853843 6221 patch_prober.go:29] interesting pod/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" start-of-body= Oct 30 02:10:52.854089 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:52.853954 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd" podUID="61c55830-d642-4daa-8692-5f4f409b9a1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.213:15021/healthz/ready\": dial tcp 10.134.0.213:15021: connect: connection refused" Oct 30 02:10:53.797463 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:53.797432 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4" Oct 30 02:10:53.797924 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:53.797751 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4" Oct 30 02:10:53.798402 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:53.798383 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4" Oct 30 02:10:53.869193 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:53.869159 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd" Oct 30 02:10:53.871229 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:53.871189 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd" Oct 30 02:10:53.872943 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:10:53.872919 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd" Oct 30 02:11:20.062731 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:20.062698 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-75bdfjd6z"] Oct 30 02:11:21.496921 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:11:21.496884 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 02:11:25.063759 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:25.063718 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-75bdfjd6z" podUID="08205ca3-93a4-4353-971e-4614a6c9ccd8" containerName="payload-processing" containerID="cri-o://2715450e5fb302e13280f911d856c9666d96d173f41b28eca6c5e63f39ebfcc2" gracePeriod=25 Oct 30 02:11:25.364149 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:25.364124 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-75bdfjd6z" Oct 30 02:11:25.398988 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:25.398962 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08205ca3-93a4-4353-971e-4614a6c9ccd8-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/08205ca3-93a4-4353-971e-4614a6c9ccd8-routing-overlay-volume\") pod \"08205ca3-93a4-4353-971e-4614a6c9ccd8\" (UID: \"08205ca3-93a4-4353-971e-4614a6c9ccd8\") " Oct 30 02:11:25.399156 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:25.399019 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08205ca3-93a4-4353-971e-4614a6c9ccd8-kube-api-access-zfp48\" (UniqueName: \"kubernetes.io/projected/08205ca3-93a4-4353-971e-4614a6c9ccd8-kube-api-access-zfp48\") pod \"08205ca3-93a4-4353-971e-4614a6c9ccd8\" (UID: \"08205ca3-93a4-4353-971e-4614a6c9ccd8\") " Oct 30 02:11:25.399156 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:25.399126 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08205ca3-93a4-4353-971e-4614a6c9ccd8-provider-credentials\" (UniqueName: \"kubernetes.io/projected/08205ca3-93a4-4353-971e-4614a6c9ccd8-provider-credentials\") pod \"08205ca3-93a4-4353-971e-4614a6c9ccd8\" (UID: \"08205ca3-93a4-4353-971e-4614a6c9ccd8\") " Oct 30 02:11:25.399156 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:25.399150 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08205ca3-93a4-4353-971e-4614a6c9ccd8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08205ca3-93a4-4353-971e-4614a6c9ccd8-plugins-config-volume\") pod \"08205ca3-93a4-4353-971e-4614a6c9ccd8\" (UID: \"08205ca3-93a4-4353-971e-4614a6c9ccd8\") " Oct 30 02:11:25.399342 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:25.399295 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08205ca3-93a4-4353-971e-4614a6c9ccd8-routing-overlay-volume" pod "08205ca3-93a4-4353-971e-4614a6c9ccd8" (UID: "08205ca3-93a4-4353-971e-4614a6c9ccd8"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:11:25.399630 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:25.399606 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08205ca3-93a4-4353-971e-4614a6c9ccd8-plugins-config-volume" pod "08205ca3-93a4-4353-971e-4614a6c9ccd8" (UID: "08205ca3-93a4-4353-971e-4614a6c9ccd8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:11:25.399904 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:25.399884 6221 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/08205ca3-93a4-4353-971e-4614a6c9ccd8-routing-overlay-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:11:25.399999 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:25.399908 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08205ca3-93a4-4353-971e-4614a6c9ccd8-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:11:25.401380 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:25.401351 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08205ca3-93a4-4353-971e-4614a6c9ccd8-provider-credentials" pod "08205ca3-93a4-4353-971e-4614a6c9ccd8" (UID: "08205ca3-93a4-4353-971e-4614a6c9ccd8"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:11:25.401471 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:25.401383 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08205ca3-93a4-4353-971e-4614a6c9ccd8-kube-api-access-zfp48" pod "08205ca3-93a4-4353-971e-4614a6c9ccd8" (UID: "08205ca3-93a4-4353-971e-4614a6c9ccd8"). InnerVolumeSpecName "kube-api-access-zfp48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:11:25.501011 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:25.500979 6221 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/08205ca3-93a4-4353-971e-4614a6c9ccd8-provider-credentials\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:11:25.501011 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:25.501006 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfp48\" (UniqueName: \"kubernetes.io/projected/08205ca3-93a4-4353-971e-4614a6c9ccd8-kube-api-access-zfp48\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:11:25.878569 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:25.878497 6221 generic.go:356] "Generic (PLEG): container finished" podID="08205ca3-93a4-4353-971e-4614a6c9ccd8" containerID="2715450e5fb302e13280f911d856c9666d96d173f41b28eca6c5e63f39ebfcc2" exitCode=0 Oct 30 02:11:25.878569 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:25.878555 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-75bdfjd6z" Oct 30 02:11:25.878569 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:25.878561 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-75bdfjd6z" event={"ID":"08205ca3-93a4-4353-971e-4614a6c9ccd8","Type":"ContainerDied","Data":"2715450e5fb302e13280f911d856c9666d96d173f41b28eca6c5e63f39ebfcc2"} Oct 30 02:11:25.878888 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:25.878617 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-75bdfjd6z" event={"ID":"08205ca3-93a4-4353-971e-4614a6c9ccd8","Type":"ContainerDied","Data":"102092d0a184cbc3e3a09976abbc10f7e9a404d935d52fda077ddc1d47ea1b35"} Oct 30 02:11:25.878888 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:25.878641 6221 scope.go:122] "RemoveContainer" containerID="2715450e5fb302e13280f911d856c9666d96d173f41b28eca6c5e63f39ebfcc2" Oct 30 02:11:25.890234 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:25.890213 6221 scope.go:122] "RemoveContainer" containerID="2715450e5fb302e13280f911d856c9666d96d173f41b28eca6c5e63f39ebfcc2" Oct 30 02:11:25.890483 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:11:25.890462 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2715450e5fb302e13280f911d856c9666d96d173f41b28eca6c5e63f39ebfcc2\": container with ID starting with 2715450e5fb302e13280f911d856c9666d96d173f41b28eca6c5e63f39ebfcc2 not found: ID does not exist" containerID="2715450e5fb302e13280f911d856c9666d96d173f41b28eca6c5e63f39ebfcc2" Oct 30 02:11:25.890545 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:25.890490 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2715450e5fb302e13280f911d856c9666d96d173f41b28eca6c5e63f39ebfcc2"} err="failed to get container status \"2715450e5fb302e13280f911d856c9666d96d173f41b28eca6c5e63f39ebfcc2\": rpc error: code = NotFound desc = could not find container \"2715450e5fb302e13280f911d856c9666d96d173f41b28eca6c5e63f39ebfcc2\": container with ID starting with 2715450e5fb302e13280f911d856c9666d96d173f41b28eca6c5e63f39ebfcc2 not found: ID does not exist" Oct 30 02:11:25.935501 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:25.935474 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-75bdfjd6z"] Oct 30 02:11:25.945668 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:25.945643 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-f1edaf/payload-processing-external-model-e2e-extmod-b-f1edaf-75bdfjd6z"] Oct 30 02:11:26.549235 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:26.549201 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08205ca3-93a4-4353-971e-4614a6c9ccd8" path="/var/lib/kubelet/pods/08205ca3-93a4-4353-971e-4614a6c9ccd8/volumes" Oct 30 02:11:33.706323 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:33.706290 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-340ae6-8f55557c5-c68h9"] Oct 30 02:11:33.707275 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:33.707255 6221 cpu_manager.go:414] "containerMap: removing container" podUID="08205ca3-93a4-4353-971e-4614a6c9ccd8" containerName="payload-processing" Oct 30 02:11:33.707275 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:33.707275 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08205ca3-93a4-4353-971e-4614a6c9ccd8" containerName="payload-processing" Oct 30 02:11:33.707399 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:33.707393 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="08205ca3-93a4-4353-971e-4614a6c9ccd8" containerName="payload-processing" Oct 30 02:11:33.711374 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:33.711357 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-340ae6-8f55557c5-c68h9" Oct 30 02:11:33.727100 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:33.727073 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-340ae6-8f55557c5-c68h9"] Oct 30 02:11:33.780552 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:33.767947 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-340ae6-dockercfg-d9z9d\"" Oct 30 02:11:33.785557 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:33.781034 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-340ae6\"" Oct 30 02:11:34.016497 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:34.016397 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc95feca-80e8-41ac-8a29-be0b9c840d1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-340ae6-8f55557c5-c68h9\" (UID: \"cc95feca-80e8-41ac-8a29-be0b9c840d1d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-340ae6-8f55557c5-c68h9" Oct 30 02:11:34.016688 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:34.016601 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjg7r\" (UniqueName: \"kubernetes.io/projected/cc95feca-80e8-41ac-8a29-be0b9c840d1d-kube-api-access-mjg7r\") pod \"payload-pre-processing-e2e-inf-a-340ae6-8f55557c5-c68h9\" (UID: \"cc95feca-80e8-41ac-8a29-be0b9c840d1d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-340ae6-8f55557c5-c68h9" Oct 30 02:11:34.118065 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:34.118022 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjg7r\" (UniqueName: \"kubernetes.io/projected/cc95feca-80e8-41ac-8a29-be0b9c840d1d-kube-api-access-mjg7r\") pod \"payload-pre-processing-e2e-inf-a-340ae6-8f55557c5-c68h9\" (UID: \"cc95feca-80e8-41ac-8a29-be0b9c840d1d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-340ae6-8f55557c5-c68h9" Oct 30 02:11:34.118346 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:34.118329 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc95feca-80e8-41ac-8a29-be0b9c840d1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-340ae6-8f55557c5-c68h9\" (UID: \"cc95feca-80e8-41ac-8a29-be0b9c840d1d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-340ae6-8f55557c5-c68h9" Oct 30 02:11:34.119067 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:34.119048 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc95feca-80e8-41ac-8a29-be0b9c840d1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-340ae6-8f55557c5-c68h9\" (UID: \"cc95feca-80e8-41ac-8a29-be0b9c840d1d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-340ae6-8f55557c5-c68h9" Oct 30 02:11:34.137760 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:34.137731 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjg7r\" (UniqueName: \"kubernetes.io/projected/cc95feca-80e8-41ac-8a29-be0b9c840d1d-kube-api-access-mjg7r\") pod \"payload-pre-processing-e2e-inf-a-340ae6-8f55557c5-c68h9\" (UID: \"cc95feca-80e8-41ac-8a29-be0b9c840d1d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-340ae6-8f55557c5-c68h9" Oct 30 02:11:34.340894 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:34.340853 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-340ae6-8f55557c5-c68h9" Oct 30 02:11:34.687789 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:34.687744 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-340ae6-8f55557c5-c68h9"] Oct 30 02:11:35.059749 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:35.059713 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-340ae6-8f55557c5-c68h9" event={"ID":"cc95feca-80e8-41ac-8a29-be0b9c840d1d","Type":"ContainerStarted","Data":"6062f35df1eabd36ad3cf281db91d864e1c1cc759bf12b732fde5a74cc85ac07"} Oct 30 02:11:35.060197 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:35.059766 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-340ae6-8f55557c5-c68h9" event={"ID":"cc95feca-80e8-41ac-8a29-be0b9c840d1d","Type":"ContainerStarted","Data":"15ca870a50a1926fe1529ecdd5a72018dc873d2cd40c81b4f090d165c4e9753d"} Oct 30 02:11:35.060197 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:35.059929 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-340ae6-8f55557c5-c68h9" Oct 30 02:11:35.101898 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:35.101831 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-340ae6-8f55557c5-c68h9" podStartSLOduration=2.101817633 podStartE2EDuration="2.101817633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:11:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:11:35.10011649 +0000 UTC m=+3218.678403822" watchObservedRunningTime="2026-10-30 02:11:35.101817633 +0000 UTC m=+3218.680104987" Oct 30 02:11:36.299672 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:36.299637 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 02:11:46.070653 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:46.070622 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-340ae6-8f55557c5-c68h9" Oct 30 02:11:51.495862 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:11:51.495827 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 02:11:53.406956 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:53.406922 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k"] Oct 30 02:11:53.413315 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:53.413292 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k" Oct 30 02:11:53.451932 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:53.451900 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-f81908-openshift-default-dockercfg-8ck82\"" Oct 30 02:11:53.458312 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:53.458286 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k"] Oct 30 02:11:53.472714 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:53.472683 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/066ee435-d202-448a-ad09-17142e92ae5f-istio-envoy\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k\" (UID: \"066ee435-d202-448a-ad09-17142e92ae5f\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k" Oct 30 02:11:53.472916 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:53.472719 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/066ee435-d202-448a-ad09-17142e92ae5f-workload-socket\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k\" (UID: \"066ee435-d202-448a-ad09-17142e92ae5f\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k" Oct 30 02:11:53.472916 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:53.472758 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/066ee435-d202-448a-ad09-17142e92ae5f-istio-data\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k\" (UID: \"066ee435-d202-448a-ad09-17142e92ae5f\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k" Oct 30 02:11:53.472916 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:53.472881 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/066ee435-d202-448a-ad09-17142e92ae5f-istio-podinfo\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k\" (UID: \"066ee435-d202-448a-ad09-17142e92ae5f\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k" Oct 30 02:11:53.473132 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:53.472980 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/066ee435-d202-448a-ad09-17142e92ae5f-workload-certs\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k\" (UID: \"066ee435-d202-448a-ad09-17142e92ae5f\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k" Oct 30 02:11:53.473132 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:53.473012 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/066ee435-d202-448a-ad09-17142e92ae5f-credential-socket\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k\" (UID: \"066ee435-d202-448a-ad09-17142e92ae5f\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k" Oct 30 02:11:53.473132 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:53.473028 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8wdn\" (UniqueName: \"kubernetes.io/projected/066ee435-d202-448a-ad09-17142e92ae5f-kube-api-access-m8wdn\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k\" (UID: \"066ee435-d202-448a-ad09-17142e92ae5f\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k" Oct 30 02:11:53.473311 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:53.473134 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/066ee435-d202-448a-ad09-17142e92ae5f-istiod-ca-cert\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k\" (UID: \"066ee435-d202-448a-ad09-17142e92ae5f\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k" Oct 30 02:11:53.473311 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:53.473228 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/066ee435-d202-448a-ad09-17142e92ae5f-istio-token\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k\" (UID: \"066ee435-d202-448a-ad09-17142e92ae5f\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k" Oct 30 02:11:53.574854 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:53.574821 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/066ee435-d202-448a-ad09-17142e92ae5f-istiod-ca-cert\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k\" (UID: \"066ee435-d202-448a-ad09-17142e92ae5f\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k" Oct 30 02:11:53.575030 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:53.574904 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/066ee435-d202-448a-ad09-17142e92ae5f-istio-token\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k\" (UID: \"066ee435-d202-448a-ad09-17142e92ae5f\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k" Oct 30 02:11:53.575030 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:53.574969 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/066ee435-d202-448a-ad09-17142e92ae5f-istio-envoy\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k\" (UID: \"066ee435-d202-448a-ad09-17142e92ae5f\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k" Oct 30 02:11:53.575030 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:53.574987 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/066ee435-d202-448a-ad09-17142e92ae5f-workload-socket\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k\" (UID: \"066ee435-d202-448a-ad09-17142e92ae5f\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k" Oct 30 02:11:53.575030 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:53.575020 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/066ee435-d202-448a-ad09-17142e92ae5f-istio-data\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k\" (UID: \"066ee435-d202-448a-ad09-17142e92ae5f\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k" Oct 30 02:11:53.575312 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:53.575110 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/066ee435-d202-448a-ad09-17142e92ae5f-istio-podinfo\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k\" (UID: \"066ee435-d202-448a-ad09-17142e92ae5f\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k" Oct 30 02:11:53.575312 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:53.575230 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/066ee435-d202-448a-ad09-17142e92ae5f-workload-certs\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k\" (UID: \"066ee435-d202-448a-ad09-17142e92ae5f\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k" Oct 30 02:11:53.575312 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:53.575282 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/066ee435-d202-448a-ad09-17142e92ae5f-credential-socket\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k\" (UID: \"066ee435-d202-448a-ad09-17142e92ae5f\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k" Oct 30 02:11:53.575503 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:53.575313 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8wdn\" (UniqueName: \"kubernetes.io/projected/066ee435-d202-448a-ad09-17142e92ae5f-kube-api-access-m8wdn\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k\" (UID: \"066ee435-d202-448a-ad09-17142e92ae5f\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k" Oct 30 02:11:53.575503 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:53.575460 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/066ee435-d202-448a-ad09-17142e92ae5f-workload-socket\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k\" (UID: \"066ee435-d202-448a-ad09-17142e92ae5f\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k" Oct 30 02:11:53.575724 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:53.575700 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/066ee435-d202-448a-ad09-17142e92ae5f-istiod-ca-cert\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k\" (UID: \"066ee435-d202-448a-ad09-17142e92ae5f\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k" Oct 30 02:11:53.575833 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:53.575792 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/066ee435-d202-448a-ad09-17142e92ae5f-istio-data\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k\" (UID: \"066ee435-d202-448a-ad09-17142e92ae5f\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k" Oct 30 02:11:53.575885 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:53.575830 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/066ee435-d202-448a-ad09-17142e92ae5f-credential-socket\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k\" (UID: \"066ee435-d202-448a-ad09-17142e92ae5f\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k" Oct 30 02:11:53.575952 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:53.575929 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/066ee435-d202-448a-ad09-17142e92ae5f-workload-certs\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k\" (UID: \"066ee435-d202-448a-ad09-17142e92ae5f\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k" Oct 30 02:11:53.577427 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:53.577397 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/066ee435-d202-448a-ad09-17142e92ae5f-istio-envoy\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k\" (UID: \"066ee435-d202-448a-ad09-17142e92ae5f\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k" Oct 30 02:11:53.577682 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:53.577665 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/066ee435-d202-448a-ad09-17142e92ae5f-istio-podinfo\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k\" (UID: \"066ee435-d202-448a-ad09-17142e92ae5f\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k" Oct 30 02:11:53.595730 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:53.595701 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/066ee435-d202-448a-ad09-17142e92ae5f-istio-token\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k\" (UID: \"066ee435-d202-448a-ad09-17142e92ae5f\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k" Oct 30 02:11:53.596368 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:53.596346 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8wdn\" (UniqueName: \"kubernetes.io/projected/066ee435-d202-448a-ad09-17142e92ae5f-kube-api-access-m8wdn\") pod \"e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k\" (UID: \"066ee435-d202-448a-ad09-17142e92ae5f\") " pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k" Oct 30 02:11:53.729664 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:53.729584 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k" Oct 30 02:11:54.056590 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:54.056565 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k"] Oct 30 02:11:54.058064 ip-10-0-135-65 kubenswrapper[6221]: W1030 02:11:54.058028 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod066ee435_d202_448a_ad09_17142e92ae5f.slice/crio-8f1d5cc85ee9c8b47ab994945db34470ccbdc3f4833feb634f5e754e25c5a90c WatchSource:0}: Error finding container 8f1d5cc85ee9c8b47ab994945db34470ccbdc3f4833feb634f5e754e25c5a90c: Status 404 returned error can't find the container with id 8f1d5cc85ee9c8b47ab994945db34470ccbdc3f4833feb634f5e754e25c5a90c Oct 30 02:11:54.061166 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:54.061133 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:11:54.061235 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:54.061201 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:11:54.061235 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:54.061231 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:11:54.279926 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:54.279889 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k" event={"ID":"066ee435-d202-448a-ad09-17142e92ae5f","Type":"ContainerStarted","Data":"395646c3dfd949809591a7ab2c58cf8f02bead0e35512c49410cdc0ec03b110b"} Oct 30 02:11:54.280102 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:54.279953 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k" event={"ID":"066ee435-d202-448a-ad09-17142e92ae5f","Type":"ContainerStarted","Data":"8f1d5cc85ee9c8b47ab994945db34470ccbdc3f4833feb634f5e754e25c5a90c"} Oct 30 02:11:54.335308 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:54.332561 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k" podStartSLOduration=1.332522384 podStartE2EDuration="1.332522384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:11:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:11:54.326767201 +0000 UTC m=+3237.905054537" watchObservedRunningTime="2026-10-30 02:11:54.332522384 +0000 UTC m=+3237.910809722" Oct 30 02:11:54.731616 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:54.730655 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k" Oct 30 02:11:55.731889 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:55.731855 6221 patch_prober.go:29] interesting pod/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Oct 30 02:11:55.732389 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:55.731907 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k" podUID="066ee435-d202-448a-ad09-17142e92ae5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Oct 30 02:11:56.730965 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:56.730927 6221 patch_prober.go:29] interesting pod/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Oct 30 02:11:56.731256 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:56.731224 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k" podUID="066ee435-d202-448a-ad09-17142e92ae5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Oct 30 02:11:57.731200 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:57.731165 6221 patch_prober.go:29] interesting pod/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Oct 30 02:11:57.731860 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:57.731819 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k" podUID="066ee435-d202-448a-ad09-17142e92ae5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Oct 30 02:11:58.731558 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:58.730885 6221 patch_prober.go:29] interesting pod/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Oct 30 02:11:58.731558 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:58.730960 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k" podUID="066ee435-d202-448a-ad09-17142e92ae5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Oct 30 02:11:59.731893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:59.731853 6221 patch_prober.go:29] interesting pod/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Oct 30 02:11:59.732426 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:11:59.731926 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k" podUID="066ee435-d202-448a-ad09-17142e92ae5f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Oct 30 02:12:00.749318 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:12:00.749283 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k" Oct 30 02:12:00.750078 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:12:00.749703 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k" Oct 30 02:12:00.750935 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:12:00.750915 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k" Oct 30 02:12:21.495895 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:12:21.495808 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 02:12:35.264837 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:12:35.264794 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f81908/test-model-f81908-kserve-7c685775d7-szqcx"] Oct 30 02:12:46.106655 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:12:46.106621 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-717065-69d69c879f-lxc5m"] Oct 30 02:12:50.292482 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:12:50.292445 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-f81908/test-model-f81908-kserve-7c685775d7-szqcx" podUID="75b98edc-c54c-48c8-a6c2-b48f7d39f7be" containerName="main" containerID="cri-o://737ce7f70a023ef4b440bf54f8866bb6f0af82fe6e8bb1ed2a8c5c36bb7a8fb0" gracePeriod=45 Oct 30 02:12:50.766867 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:12:50.766837 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f81908/test-model-f81908-kserve-7c685775d7-szqcx" Oct 30 02:12:50.886591 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:12:50.886469 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75b98edc-c54c-48c8-a6c2-b48f7d39f7be-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75b98edc-c54c-48c8-a6c2-b48f7d39f7be-tmp-dir\") pod \"75b98edc-c54c-48c8-a6c2-b48f7d39f7be\" (UID: \"75b98edc-c54c-48c8-a6c2-b48f7d39f7be\") " Oct 30 02:12:50.886780 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:12:50.886646 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75b98edc-c54c-48c8-a6c2-b48f7d39f7be-kube-api-access-jh2dx\" (UniqueName: \"kubernetes.io/projected/75b98edc-c54c-48c8-a6c2-b48f7d39f7be-kube-api-access-jh2dx\") pod \"75b98edc-c54c-48c8-a6c2-b48f7d39f7be\" (UID: \"75b98edc-c54c-48c8-a6c2-b48f7d39f7be\") " Oct 30 02:12:50.886780 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:12:50.886687 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75b98edc-c54c-48c8-a6c2-b48f7d39f7be-dshm\" (UniqueName: \"kubernetes.io/empty-dir/75b98edc-c54c-48c8-a6c2-b48f7d39f7be-dshm\") pod \"75b98edc-c54c-48c8-a6c2-b48f7d39f7be\" (UID: \"75b98edc-c54c-48c8-a6c2-b48f7d39f7be\") " Oct 30 02:12:50.886780 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:12:50.886751 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75b98edc-c54c-48c8-a6c2-b48f7d39f7be-home\" (UniqueName: \"kubernetes.io/empty-dir/75b98edc-c54c-48c8-a6c2-b48f7d39f7be-home\") pod \"75b98edc-c54c-48c8-a6c2-b48f7d39f7be\" (UID: \"75b98edc-c54c-48c8-a6c2-b48f7d39f7be\") " Oct 30 02:12:50.886780 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:12:50.886761 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75b98edc-c54c-48c8-a6c2-b48f7d39f7be-tmp-dir" pod "75b98edc-c54c-48c8-a6c2-b48f7d39f7be" (UID: "75b98edc-c54c-48c8-a6c2-b48f7d39f7be"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:12:50.887029 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:12:50.886852 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75b98edc-c54c-48c8-a6c2-b48f7d39f7be-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75b98edc-c54c-48c8-a6c2-b48f7d39f7be-model-cache\") pod \"75b98edc-c54c-48c8-a6c2-b48f7d39f7be\" (UID: \"75b98edc-c54c-48c8-a6c2-b48f7d39f7be\") " Oct 30 02:12:50.887086 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:12:50.887045 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75b98edc-c54c-48c8-a6c2-b48f7d39f7be-tls-certs\" (UniqueName: \"kubernetes.io/secret/75b98edc-c54c-48c8-a6c2-b48f7d39f7be-tls-certs\") pod \"75b98edc-c54c-48c8-a6c2-b48f7d39f7be\" (UID: \"75b98edc-c54c-48c8-a6c2-b48f7d39f7be\") " Oct 30 02:12:50.887324 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:12:50.887301 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75b98edc-c54c-48c8-a6c2-b48f7d39f7be-home" pod "75b98edc-c54c-48c8-a6c2-b48f7d39f7be" (UID: "75b98edc-c54c-48c8-a6c2-b48f7d39f7be"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:12:50.887741 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:12:50.887723 6221 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75b98edc-c54c-48c8-a6c2-b48f7d39f7be-tmp-dir\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:12:50.887844 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:12:50.887744 6221 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75b98edc-c54c-48c8-a6c2-b48f7d39f7be-home\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:12:50.888163 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:12:50.888144 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75b98edc-c54c-48c8-a6c2-b48f7d39f7be-model-cache" pod "75b98edc-c54c-48c8-a6c2-b48f7d39f7be" (UID: "75b98edc-c54c-48c8-a6c2-b48f7d39f7be"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:12:50.893589 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:12:50.892617 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75b98edc-c54c-48c8-a6c2-b48f7d39f7be-dshm" pod "75b98edc-c54c-48c8-a6c2-b48f7d39f7be" (UID: "75b98edc-c54c-48c8-a6c2-b48f7d39f7be"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:12:50.897987 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:12:50.897952 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75b98edc-c54c-48c8-a6c2-b48f7d39f7be-kube-api-access-jh2dx" pod "75b98edc-c54c-48c8-a6c2-b48f7d39f7be" (UID: "75b98edc-c54c-48c8-a6c2-b48f7d39f7be"). InnerVolumeSpecName "kube-api-access-jh2dx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:12:50.899869 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:12:50.899845 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75b98edc-c54c-48c8-a6c2-b48f7d39f7be-tls-certs" pod "75b98edc-c54c-48c8-a6c2-b48f7d39f7be" (UID: "75b98edc-c54c-48c8-a6c2-b48f7d39f7be"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:12:50.915986 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:12:50.915948 6221 generic.go:356] "Generic (PLEG): container finished" podID="75b98edc-c54c-48c8-a6c2-b48f7d39f7be" containerID="737ce7f70a023ef4b440bf54f8866bb6f0af82fe6e8bb1ed2a8c5c36bb7a8fb0" exitCode=0 Oct 30 02:12:50.916167 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:12:50.916014 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f81908/test-model-f81908-kserve-7c685775d7-szqcx" Oct 30 02:12:50.916238 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:12:50.916187 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f81908/test-model-f81908-kserve-7c685775d7-szqcx" event={"ID":"75b98edc-c54c-48c8-a6c2-b48f7d39f7be","Type":"ContainerDied","Data":"737ce7f70a023ef4b440bf54f8866bb6f0af82fe6e8bb1ed2a8c5c36bb7a8fb0"} Oct 30 02:12:50.916238 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:12:50.916221 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f81908/test-model-f81908-kserve-7c685775d7-szqcx" event={"ID":"75b98edc-c54c-48c8-a6c2-b48f7d39f7be","Type":"ContainerDied","Data":"7cb6b58cc22706cf6fa4a840ad354ca874e150643be8bf6a511fec760930f32a"} Oct 30 02:12:50.916332 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:12:50.916242 6221 scope.go:122] "RemoveContainer" containerID="737ce7f70a023ef4b440bf54f8866bb6f0af82fe6e8bb1ed2a8c5c36bb7a8fb0" Oct 30 02:12:50.940806 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:12:50.940783 6221 scope.go:122] "RemoveContainer" containerID="737ce7f70a023ef4b440bf54f8866bb6f0af82fe6e8bb1ed2a8c5c36bb7a8fb0" Oct 30 02:12:50.941133 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:12:50.941107 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"737ce7f70a023ef4b440bf54f8866bb6f0af82fe6e8bb1ed2a8c5c36bb7a8fb0\": container with ID starting with 737ce7f70a023ef4b440bf54f8866bb6f0af82fe6e8bb1ed2a8c5c36bb7a8fb0 not found: ID does not exist" containerID="737ce7f70a023ef4b440bf54f8866bb6f0af82fe6e8bb1ed2a8c5c36bb7a8fb0" Oct 30 02:12:50.941243 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:12:50.941139 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"737ce7f70a023ef4b440bf54f8866bb6f0af82fe6e8bb1ed2a8c5c36bb7a8fb0"} err="failed to get container status \"737ce7f70a023ef4b440bf54f8866bb6f0af82fe6e8bb1ed2a8c5c36bb7a8fb0\": rpc error: code = NotFound desc = could not find container \"737ce7f70a023ef4b440bf54f8866bb6f0af82fe6e8bb1ed2a8c5c36bb7a8fb0\": container with ID starting with 737ce7f70a023ef4b440bf54f8866bb6f0af82fe6e8bb1ed2a8c5c36bb7a8fb0 not found: ID does not exist" Oct 30 02:12:50.989419 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:12:50.989373 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f81908/test-model-f81908-kserve-7c685775d7-szqcx"] Oct 30 02:12:50.991078 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:12:50.991042 6221 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75b98edc-c54c-48c8-a6c2-b48f7d39f7be-tls-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:12:50.991078 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:12:50.991080 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh2dx\" (UniqueName: \"kubernetes.io/projected/75b98edc-c54c-48c8-a6c2-b48f7d39f7be-kube-api-access-jh2dx\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:12:50.991292 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:12:50.991095 6221 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75b98edc-c54c-48c8-a6c2-b48f7d39f7be-dshm\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:12:50.991292 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:12:50.991109 6221 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75b98edc-c54c-48c8-a6c2-b48f7d39f7be-model-cache\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:12:50.996609 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:12:50.996576 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-f81908/test-model-f81908-kserve-7c685775d7-szqcx"] Oct 30 02:12:51.107402 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:12:51.107363 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-717065-69d69c879f-lxc5m" podUID="5b74befa-31a1-4198-80a6-a46db3294eaa" containerName="payload-pre-processing" containerID="cri-o://fae51e8a708bd5488ca610e54f9e390cd9cbc40faa6ee850494e4e3492e4b6f7" gracePeriod=25 Oct 30 02:12:51.495632 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:12:51.495596 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 02:12:52.553349 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:12:52.553315 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75b98edc-c54c-48c8-a6c2-b48f7d39f7be" path="/var/lib/kubelet/pods/75b98edc-c54c-48c8-a6c2-b48f7d39f7be/volumes" Oct 30 02:12:53.286262 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:12:53.286216 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-717065-69d69c879f-lxc5m" podUID="5b74befa-31a1-4198-80a6-a46db3294eaa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:12:57.234774 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:12:57.234740 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2vsns_d332bbb4-c699-44e8-96ba-516d8ca03e95/ovn-acl-logging/0.log" Oct 30 02:12:57.236672 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:12:57.236648 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2vsns_d332bbb4-c699-44e8-96ba-516d8ca03e95/ovn-acl-logging/0.log" Oct 30 02:13:03.086884 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:03.086845 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-717065-7d598477fd-zrzd7"] Oct 30 02:13:03.087479 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:03.087129 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-717065-7d598477fd-zrzd7" podUID="e4d33590-0112-4d25-b383-a7b6757d480d" containerName="maas-api" containerID="cri-o://28fd6d46905190e83831dc0fc83c1822908f2fd89da34e6c23662903a24c49a6" gracePeriod=30 Oct 30 02:13:03.286792 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:03.286749 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-717065-69d69c879f-lxc5m" podUID="5b74befa-31a1-4198-80a6-a46db3294eaa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:13:03.662554 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:03.660904 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-717065-7d598477fd-zrzd7" podUID="e4d33590-0112-4d25-b383-a7b6757d480d" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.197:8443/health\": dial tcp 10.134.0.197:8443: connect: connection refused" Oct 30 02:13:04.378453 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:04.378422 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-717065-7d598477fd-zrzd7" Oct 30 02:13:04.420048 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:04.419956 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4d33590-0112-4d25-b383-a7b6757d480d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4d33590-0112-4d25-b383-a7b6757d480d-openshift-service-ca\") pod \"e4d33590-0112-4d25-b383-a7b6757d480d\" (UID: \"e4d33590-0112-4d25-b383-a7b6757d480d\") " Oct 30 02:13:04.420241 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:04.420188 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4d33590-0112-4d25-b383-a7b6757d480d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4d33590-0112-4d25-b383-a7b6757d480d-metrics-certs\") pod \"e4d33590-0112-4d25-b383-a7b6757d480d\" (UID: \"e4d33590-0112-4d25-b383-a7b6757d480d\") " Oct 30 02:13:04.421063 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:04.420365 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4d33590-0112-4d25-b383-a7b6757d480d-kube-api-access-9zgch\" (UniqueName: \"kubernetes.io/projected/e4d33590-0112-4d25-b383-a7b6757d480d-kube-api-access-9zgch\") pod \"e4d33590-0112-4d25-b383-a7b6757d480d\" (UID: \"e4d33590-0112-4d25-b383-a7b6757d480d\") " Oct 30 02:13:04.421063 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:04.420407 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4d33590-0112-4d25-b383-a7b6757d480d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4d33590-0112-4d25-b383-a7b6757d480d-maas-api-tls\") pod \"e4d33590-0112-4d25-b383-a7b6757d480d\" (UID: \"e4d33590-0112-4d25-b383-a7b6757d480d\") " Oct 30 02:13:04.421063 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:04.420486 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4d33590-0112-4d25-b383-a7b6757d480d-openshift-service-ca" pod "e4d33590-0112-4d25-b383-a7b6757d480d" (UID: "e4d33590-0112-4d25-b383-a7b6757d480d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:13:04.421063 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:04.421034 6221 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e4d33590-0112-4d25-b383-a7b6757d480d-openshift-service-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:13:04.424075 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:04.424023 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4d33590-0112-4d25-b383-a7b6757d480d-maas-api-tls" pod "e4d33590-0112-4d25-b383-a7b6757d480d" (UID: "e4d33590-0112-4d25-b383-a7b6757d480d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:13:04.429574 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:04.427642 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4d33590-0112-4d25-b383-a7b6757d480d-kube-api-access-9zgch" pod "e4d33590-0112-4d25-b383-a7b6757d480d" (UID: "e4d33590-0112-4d25-b383-a7b6757d480d"). InnerVolumeSpecName "kube-api-access-9zgch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:13:04.433579 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:04.432089 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4d33590-0112-4d25-b383-a7b6757d480d-metrics-certs" pod "e4d33590-0112-4d25-b383-a7b6757d480d" (UID: "e4d33590-0112-4d25-b383-a7b6757d480d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:13:04.522342 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:04.522303 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zgch\" (UniqueName: \"kubernetes.io/projected/e4d33590-0112-4d25-b383-a7b6757d480d-kube-api-access-9zgch\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:13:04.522342 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:04.522337 6221 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e4d33590-0112-4d25-b383-a7b6757d480d-maas-api-tls\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:13:04.522342 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:04.522347 6221 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e4d33590-0112-4d25-b383-a7b6757d480d-metrics-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:13:05.029883 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:05.029838 6221 generic.go:356] "Generic (PLEG): container finished" podID="e4d33590-0112-4d25-b383-a7b6757d480d" containerID="28fd6d46905190e83831dc0fc83c1822908f2fd89da34e6c23662903a24c49a6" exitCode=0 Oct 30 02:13:05.030112 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:05.029895 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-717065-7d598477fd-zrzd7" event={"ID":"e4d33590-0112-4d25-b383-a7b6757d480d","Type":"ContainerDied","Data":"28fd6d46905190e83831dc0fc83c1822908f2fd89da34e6c23662903a24c49a6"} Oct 30 02:13:05.030112 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:05.029914 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-717065-7d598477fd-zrzd7" Oct 30 02:13:05.030112 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:05.029954 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-717065-7d598477fd-zrzd7" event={"ID":"e4d33590-0112-4d25-b383-a7b6757d480d","Type":"ContainerDied","Data":"5e95bd553cf556b2b814ba6ceac39b228447b1ceb9f3a7f524d60566ed9636e8"} Oct 30 02:13:05.030112 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:05.029979 6221 scope.go:122] "RemoveContainer" containerID="28fd6d46905190e83831dc0fc83c1822908f2fd89da34e6c23662903a24c49a6" Oct 30 02:13:05.041012 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:05.040988 6221 scope.go:122] "RemoveContainer" containerID="28fd6d46905190e83831dc0fc83c1822908f2fd89da34e6c23662903a24c49a6" Oct 30 02:13:05.041313 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:13:05.041288 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28fd6d46905190e83831dc0fc83c1822908f2fd89da34e6c23662903a24c49a6\": container with ID starting with 28fd6d46905190e83831dc0fc83c1822908f2fd89da34e6c23662903a24c49a6 not found: ID does not exist" containerID="28fd6d46905190e83831dc0fc83c1822908f2fd89da34e6c23662903a24c49a6" Oct 30 02:13:05.041435 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:05.041320 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28fd6d46905190e83831dc0fc83c1822908f2fd89da34e6c23662903a24c49a6"} err="failed to get container status \"28fd6d46905190e83831dc0fc83c1822908f2fd89da34e6c23662903a24c49a6\": rpc error: code = NotFound desc = could not find container \"28fd6d46905190e83831dc0fc83c1822908f2fd89da34e6c23662903a24c49a6\": container with ID starting with 28fd6d46905190e83831dc0fc83c1822908f2fd89da34e6c23662903a24c49a6 not found: ID does not exist" Oct 30 02:13:05.082070 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:05.082039 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-717065-7d598477fd-zrzd7"] Oct 30 02:13:05.094290 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:05.094260 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-717065-7d598477fd-zrzd7"] Oct 30 02:13:06.549732 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:06.549699 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4d33590-0112-4d25-b383-a7b6757d480d" path="/var/lib/kubelet/pods/e4d33590-0112-4d25-b383-a7b6757d480d/volumes" Oct 30 02:13:06.814629 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:06.812932 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 02:13:10.910580 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:10.910526 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g"] Oct 30 02:13:10.911123 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:10.910816 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g" podUID="4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2" containerName="istio-proxy" containerID="cri-o://464258c08cc04e9e13f1f8efdffe27e93eef437969e199be187a1128c43963f7" gracePeriod=30 Oct 30 02:13:11.402104 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:11.402076 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-717065-69d69c879f-lxc5m" Oct 30 02:13:11.507904 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:11.507821 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b74befa-31a1-4198-80a6-a46db3294eaa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b74befa-31a1-4198-80a6-a46db3294eaa-plugins-config-volume\") pod \"5b74befa-31a1-4198-80a6-a46db3294eaa\" (UID: \"5b74befa-31a1-4198-80a6-a46db3294eaa\") " Oct 30 02:13:11.508063 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:11.508032 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b74befa-31a1-4198-80a6-a46db3294eaa-kube-api-access-x62mw\" (UniqueName: \"kubernetes.io/projected/5b74befa-31a1-4198-80a6-a46db3294eaa-kube-api-access-x62mw\") pod \"5b74befa-31a1-4198-80a6-a46db3294eaa\" (UID: \"5b74befa-31a1-4198-80a6-a46db3294eaa\") " Oct 30 02:13:11.508252 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:11.508224 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b74befa-31a1-4198-80a6-a46db3294eaa-plugins-config-volume" pod "5b74befa-31a1-4198-80a6-a46db3294eaa" (UID: "5b74befa-31a1-4198-80a6-a46db3294eaa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:13:11.508431 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:11.508415 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b74befa-31a1-4198-80a6-a46db3294eaa-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:13:11.510226 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:11.510199 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b74befa-31a1-4198-80a6-a46db3294eaa-kube-api-access-x62mw" pod "5b74befa-31a1-4198-80a6-a46db3294eaa" (UID: "5b74befa-31a1-4198-80a6-a46db3294eaa"). InnerVolumeSpecName "kube-api-access-x62mw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:13:11.609999 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:11.609958 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x62mw\" (UniqueName: \"kubernetes.io/projected/5b74befa-31a1-4198-80a6-a46db3294eaa-kube-api-access-x62mw\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:13:12.166803 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:12.166762 6221 generic.go:356] "Generic (PLEG): container finished" podID="5b74befa-31a1-4198-80a6-a46db3294eaa" containerID="fae51e8a708bd5488ca610e54f9e390cd9cbc40faa6ee850494e4e3492e4b6f7" exitCode=0 Oct 30 02:13:12.167271 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:12.166805 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-717065-69d69c879f-lxc5m" Oct 30 02:13:12.167271 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:12.166840 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-717065-69d69c879f-lxc5m" event={"ID":"5b74befa-31a1-4198-80a6-a46db3294eaa","Type":"ContainerDied","Data":"fae51e8a708bd5488ca610e54f9e390cd9cbc40faa6ee850494e4e3492e4b6f7"} Oct 30 02:13:12.167271 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:12.166899 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-717065-69d69c879f-lxc5m" event={"ID":"5b74befa-31a1-4198-80a6-a46db3294eaa","Type":"ContainerDied","Data":"54a9a4b9ccabb0d71156eb42cbd4ef56237976166f9e82403d8215e6796a976c"} Oct 30 02:13:12.167271 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:12.166935 6221 scope.go:122] "RemoveContainer" containerID="fae51e8a708bd5488ca610e54f9e390cd9cbc40faa6ee850494e4e3492e4b6f7" Oct 30 02:13:12.178671 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:12.178653 6221 scope.go:122] "RemoveContainer" containerID="fae51e8a708bd5488ca610e54f9e390cd9cbc40faa6ee850494e4e3492e4b6f7" Oct 30 02:13:12.178947 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:13:12.178927 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fae51e8a708bd5488ca610e54f9e390cd9cbc40faa6ee850494e4e3492e4b6f7\": container with ID starting with fae51e8a708bd5488ca610e54f9e390cd9cbc40faa6ee850494e4e3492e4b6f7 not found: ID does not exist" containerID="fae51e8a708bd5488ca610e54f9e390cd9cbc40faa6ee850494e4e3492e4b6f7" Oct 30 02:13:12.178999 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:12.178956 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fae51e8a708bd5488ca610e54f9e390cd9cbc40faa6ee850494e4e3492e4b6f7"} err="failed to get container status \"fae51e8a708bd5488ca610e54f9e390cd9cbc40faa6ee850494e4e3492e4b6f7\": rpc error: code = NotFound desc = could not find container \"fae51e8a708bd5488ca610e54f9e390cd9cbc40faa6ee850494e4e3492e4b6f7\": container with ID starting with fae51e8a708bd5488ca610e54f9e390cd9cbc40faa6ee850494e4e3492e4b6f7 not found: ID does not exist" Oct 30 02:13:12.221167 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:12.221133 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-717065-69d69c879f-lxc5m"] Oct 30 02:13:12.235125 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:12.235086 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-717065-69d69c879f-lxc5m"] Oct 30 02:13:12.549755 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:12.549725 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b74befa-31a1-4198-80a6-a46db3294eaa" path="/var/lib/kubelet/pods/5b74befa-31a1-4198-80a6-a46db3294eaa/volumes" Oct 30 02:13:14.942079 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:14.942039 6221 patch_prober.go:29] interesting pod/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:13:14.942483 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:14.942112 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g" podUID="4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:13:16.206459 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:16.206351 6221 generic.go:356] "Generic (PLEG): container finished" podID="4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2" containerID="464258c08cc04e9e13f1f8efdffe27e93eef437969e199be187a1128c43963f7" exitCode=0 Oct 30 02:13:16.207147 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:16.206962 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g" event={"ID":"4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2","Type":"ContainerDied","Data":"464258c08cc04e9e13f1f8efdffe27e93eef437969e199be187a1128c43963f7"} Oct 30 02:13:16.249352 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:16.249328 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g" Oct 30 02:13:16.281930 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:16.281904 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-istio-data\") pod \"4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2\" (UID: \"4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2\") " Oct 30 02:13:16.282084 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:16.281955 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-credential-socket\") pod \"4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2\" (UID: \"4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2\") " Oct 30 02:13:16.282153 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:16.282094 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-istio-envoy\") pod \"4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2\" (UID: \"4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2\") " Oct 30 02:13:16.282153 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:16.282134 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-kube-api-access-sc6n5\" (UniqueName: \"kubernetes.io/projected/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-kube-api-access-sc6n5\") pod \"4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2\" (UID: \"4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2\") " Oct 30 02:13:16.282286 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:16.282166 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-istio-data" pod "4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2" (UID: "4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:16.282286 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:16.282245 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-workload-socket\") pod \"4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2\" (UID: \"4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2\") " Oct 30 02:13:16.282286 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:16.282273 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-credential-socket" pod "4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2" (UID: "4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:16.282458 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:16.282294 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-istio-podinfo\") pod \"4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2\" (UID: \"4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2\") " Oct 30 02:13:16.282458 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:16.282321 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-istio-token\" (UniqueName: \"kubernetes.io/projected/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-istio-token\") pod \"4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2\" (UID: \"4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2\") " Oct 30 02:13:16.282458 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:16.282417 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-workload-certs\") pod \"4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2\" (UID: \"4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2\") " Oct 30 02:13:16.282653 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:16.282456 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-istiod-ca-cert\") pod \"4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2\" (UID: \"4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2\") " Oct 30 02:13:16.282653 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:16.282499 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-workload-socket" pod "4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2" (UID: "4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:16.282769 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:16.282708 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-workload-certs" pod "4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2" (UID: "4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:16.283045 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:16.283024 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-istiod-ca-cert" pod "4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2" (UID: "4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:13:16.283296 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:16.283276 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:13:16.283383 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:16.283300 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:13:16.283383 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:16.283313 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:13:16.283383 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:16.283325 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:13:16.283383 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:16.283333 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:13:16.284955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:16.284934 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-istio-token" pod "4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2" (UID: "4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:13:16.285040 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:16.284971 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-istio-envoy" pod "4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2" (UID: "4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:13:16.285040 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:16.284985 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-istio-podinfo" pod "4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2" (UID: "4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:13:16.285040 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:16.284996 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-kube-api-access-sc6n5" pod "4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2" (UID: "4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2"). InnerVolumeSpecName "kube-api-access-sc6n5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:13:16.384593 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:16.384562 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:13:16.384593 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:16.384590 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc6n5\" (UniqueName: \"kubernetes.io/projected/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-kube-api-access-sc6n5\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:13:16.384795 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:16.384602 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:13:16.384795 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:16.384611 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:13:17.215680 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:17.215652 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g" Oct 30 02:13:17.216275 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:17.215691 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g" event={"ID":"4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2","Type":"ContainerDied","Data":"1280e83fbc02753515cf548d96c575be2e559268ffbdfb777a36936d2030c6e4"} Oct 30 02:13:17.216275 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:17.215765 6221 scope.go:122] "RemoveContainer" containerID="464258c08cc04e9e13f1f8efdffe27e93eef437969e199be187a1128c43963f7" Oct 30 02:13:17.282931 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:17.282899 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g"] Oct 30 02:13:17.295876 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:17.295842 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-717065-openshift-default-6787fd448c-62q4g"] Oct 30 02:13:18.548922 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:18.548892 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2" path="/var/lib/kubelet/pods/4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2/volumes" Oct 30 02:13:21.495239 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:13:21.495202 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 02:13:21.937973 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:21.937941 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f81908-55df8db8bb-dlxz9"] Oct 30 02:13:26.938764 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:26.938715 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-f81908-55df8db8bb-dlxz9" podUID="3af28e5b-f8a6-46b4-af05-f8ea92975c01" containerName="payload-processing" containerID="cri-o://824807c5eda587ce9b3b7a2c51fd761c2abacdce8cdc1230f74f98ce1aee6296" gracePeriod=25 Oct 30 02:13:27.044433 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:27.044397 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2d5aa9-698b999554-z8f24"] Oct 30 02:13:27.045203 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:27.045189 6221 cpu_manager.go:414] "containerMap: removing container" podUID="e4d33590-0112-4d25-b383-a7b6757d480d" containerName="maas-api" Oct 30 02:13:27.045262 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:27.045205 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4d33590-0112-4d25-b383-a7b6757d480d" containerName="maas-api" Oct 30 02:13:27.045262 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:27.045220 6221 cpu_manager.go:414] "containerMap: removing container" podUID="75b98edc-c54c-48c8-a6c2-b48f7d39f7be" containerName="main" Oct 30 02:13:27.045262 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:27.045226 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75b98edc-c54c-48c8-a6c2-b48f7d39f7be" containerName="main" Oct 30 02:13:27.045262 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:27.045233 6221 cpu_manager.go:414] "containerMap: removing container" podUID="4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2" containerName="istio-proxy" Oct 30 02:13:27.045262 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:27.045238 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2" containerName="istio-proxy" Oct 30 02:13:27.045262 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:27.045250 6221 cpu_manager.go:414] "containerMap: removing container" podUID="5b74befa-31a1-4198-80a6-a46db3294eaa" containerName="payload-pre-processing" Oct 30 02:13:27.045262 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:27.045255 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b74befa-31a1-4198-80a6-a46db3294eaa" containerName="payload-pre-processing" Oct 30 02:13:27.045493 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:27.045374 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4d33590-0112-4d25-b383-a7b6757d480d" containerName="maas-api" Oct 30 02:13:27.045493 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:27.045382 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="75b98edc-c54c-48c8-a6c2-b48f7d39f7be" containerName="main" Oct 30 02:13:27.045493 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:27.045388 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b4b729b-a3fc-421d-bca1-3b7dcfd6b0a2" containerName="istio-proxy" Oct 30 02:13:27.045493 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:27.045395 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b74befa-31a1-4198-80a6-a46db3294eaa" containerName="payload-pre-processing" Oct 30 02:13:27.050802 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:27.050776 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2d5aa9-698b999554-z8f24" Oct 30 02:13:27.066645 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:27.066619 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2d5aa9-698b999554-z8f24"] Oct 30 02:13:27.074181 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:27.074156 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-2d5aa9\"" Oct 30 02:13:27.087270 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:27.087242 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlwdv\" (UniqueName: \"kubernetes.io/projected/27948676-86ab-49f1-a4f7-3083a591738c-kube-api-access-mlwdv\") pod \"payload-processing-e2e-inf-b-2d5aa9-698b999554-z8f24\" (UID: \"27948676-86ab-49f1-a4f7-3083a591738c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2d5aa9-698b999554-z8f24" Oct 30 02:13:27.087410 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:27.087312 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27948676-86ab-49f1-a4f7-3083a591738c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2d5aa9-698b999554-z8f24\" (UID: \"27948676-86ab-49f1-a4f7-3083a591738c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2d5aa9-698b999554-z8f24" Oct 30 02:13:27.206653 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:27.198495 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlwdv\" (UniqueName: \"kubernetes.io/projected/27948676-86ab-49f1-a4f7-3083a591738c-kube-api-access-mlwdv\") pod \"payload-processing-e2e-inf-b-2d5aa9-698b999554-z8f24\" (UID: \"27948676-86ab-49f1-a4f7-3083a591738c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2d5aa9-698b999554-z8f24" Oct 30 02:13:27.206653 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:27.198591 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27948676-86ab-49f1-a4f7-3083a591738c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2d5aa9-698b999554-z8f24\" (UID: \"27948676-86ab-49f1-a4f7-3083a591738c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2d5aa9-698b999554-z8f24" Oct 30 02:13:27.254625 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:27.254582 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-f81908-55df8db8bb-dlxz9" podUID="3af28e5b-f8a6-46b4-af05-f8ea92975c01" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:13:27.256206 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:27.256179 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27948676-86ab-49f1-a4f7-3083a591738c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2d5aa9-698b999554-z8f24\" (UID: \"27948676-86ab-49f1-a4f7-3083a591738c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2d5aa9-698b999554-z8f24" Oct 30 02:13:27.266554 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:27.265191 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlwdv\" (UniqueName: \"kubernetes.io/projected/27948676-86ab-49f1-a4f7-3083a591738c-kube-api-access-mlwdv\") pod \"payload-processing-e2e-inf-b-2d5aa9-698b999554-z8f24\" (UID: \"27948676-86ab-49f1-a4f7-3083a591738c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2d5aa9-698b999554-z8f24" Oct 30 02:13:27.370551 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:27.366823 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2d5aa9-698b999554-z8f24" Oct 30 02:13:27.729703 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:27.729675 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2d5aa9-698b999554-z8f24"] Oct 30 02:13:27.732330 ip-10-0-135-65 kubenswrapper[6221]: W1030 02:13:27.732301 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27948676_86ab_49f1_a4f7_3083a591738c.slice/crio-201ee2c2b5e779e7ca863aa65aa7924ee30ffa42e1d214f954775128ae4a5017 WatchSource:0}: Error finding container 201ee2c2b5e779e7ca863aa65aa7924ee30ffa42e1d214f954775128ae4a5017: Status 404 returned error can't find the container with id 201ee2c2b5e779e7ca863aa65aa7924ee30ffa42e1d214f954775128ae4a5017 Oct 30 02:13:28.351015 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:28.350982 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2d5aa9-698b999554-z8f24" event={"ID":"27948676-86ab-49f1-a4f7-3083a591738c","Type":"ContainerStarted","Data":"c67ae1db4c5cb414f76612649870aa110c0597a074adf8d0251e7b75cb1acd98"} Oct 30 02:13:28.351437 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:28.351042 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2d5aa9-698b999554-z8f24" event={"ID":"27948676-86ab-49f1-a4f7-3083a591738c","Type":"ContainerStarted","Data":"201ee2c2b5e779e7ca863aa65aa7924ee30ffa42e1d214f954775128ae4a5017"} Oct 30 02:13:28.351437 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:28.351071 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2d5aa9-698b999554-z8f24" Oct 30 02:13:28.394772 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:28.394726 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-2d5aa9-698b999554-z8f24" podStartSLOduration=1.3947063929999999 podStartE2EDuration="1.394706393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:13:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:13:28.389838574 +0000 UTC m=+3331.968125910" watchObservedRunningTime="2026-10-30 02:13:28.394706393 +0000 UTC m=+3331.972993730" Oct 30 02:13:30.442653 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:30.442616 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-817fd7-764dbbc7f9-9fsbh"] Oct 30 02:13:30.443185 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:30.442867 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-817fd7-764dbbc7f9-9fsbh" podUID="f8de244f-5cf0-4543-9118-6684c2be86f0" containerName="maas-api" containerID="cri-o://73ad6962f718bda9b4dc6453a8ed6a38e66d6a98cd0e2813f05aae1443cf9f12" gracePeriod=30 Oct 30 02:13:31.011479 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:31.011086 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-817fd7-764dbbc7f9-9fsbh" Oct 30 02:13:31.171464 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:31.171371 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8de244f-5cf0-4543-9118-6684c2be86f0-kube-api-access-96d5w\" (UniqueName: \"kubernetes.io/projected/f8de244f-5cf0-4543-9118-6684c2be86f0-kube-api-access-96d5w\") pod \"f8de244f-5cf0-4543-9118-6684c2be86f0\" (UID: \"f8de244f-5cf0-4543-9118-6684c2be86f0\") " Oct 30 02:13:31.171464 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:31.171445 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8de244f-5cf0-4543-9118-6684c2be86f0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8de244f-5cf0-4543-9118-6684c2be86f0-maas-api-tls\") pod \"f8de244f-5cf0-4543-9118-6684c2be86f0\" (UID: \"f8de244f-5cf0-4543-9118-6684c2be86f0\") " Oct 30 02:13:31.171732 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:31.171659 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8de244f-5cf0-4543-9118-6684c2be86f0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8de244f-5cf0-4543-9118-6684c2be86f0-metrics-certs\") pod \"f8de244f-5cf0-4543-9118-6684c2be86f0\" (UID: \"f8de244f-5cf0-4543-9118-6684c2be86f0\") " Oct 30 02:13:31.171732 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:31.171703 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8de244f-5cf0-4543-9118-6684c2be86f0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8de244f-5cf0-4543-9118-6684c2be86f0-openshift-service-ca\") pod \"f8de244f-5cf0-4543-9118-6684c2be86f0\" (UID: \"f8de244f-5cf0-4543-9118-6684c2be86f0\") " Oct 30 02:13:31.172224 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:31.172191 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8de244f-5cf0-4543-9118-6684c2be86f0-openshift-service-ca" pod "f8de244f-5cf0-4543-9118-6684c2be86f0" (UID: "f8de244f-5cf0-4543-9118-6684c2be86f0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:13:31.172638 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:31.172611 6221 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8de244f-5cf0-4543-9118-6684c2be86f0-openshift-service-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:13:31.173928 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:31.173904 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8de244f-5cf0-4543-9118-6684c2be86f0-kube-api-access-96d5w" pod "f8de244f-5cf0-4543-9118-6684c2be86f0" (UID: "f8de244f-5cf0-4543-9118-6684c2be86f0"). InnerVolumeSpecName "kube-api-access-96d5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:13:31.174057 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:31.173994 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8de244f-5cf0-4543-9118-6684c2be86f0-maas-api-tls" pod "f8de244f-5cf0-4543-9118-6684c2be86f0" (UID: "f8de244f-5cf0-4543-9118-6684c2be86f0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:13:31.174207 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:31.174182 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8de244f-5cf0-4543-9118-6684c2be86f0-metrics-certs" pod "f8de244f-5cf0-4543-9118-6684c2be86f0" (UID: "f8de244f-5cf0-4543-9118-6684c2be86f0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:13:31.279094 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:31.278994 6221 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f8de244f-5cf0-4543-9118-6684c2be86f0-metrics-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:13:31.279094 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:31.279038 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96d5w\" (UniqueName: \"kubernetes.io/projected/f8de244f-5cf0-4543-9118-6684c2be86f0-kube-api-access-96d5w\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:13:31.279094 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:31.279056 6221 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f8de244f-5cf0-4543-9118-6684c2be86f0-maas-api-tls\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:13:31.375569 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:31.375518 6221 generic.go:356] "Generic (PLEG): container finished" podID="f8de244f-5cf0-4543-9118-6684c2be86f0" containerID="73ad6962f718bda9b4dc6453a8ed6a38e66d6a98cd0e2813f05aae1443cf9f12" exitCode=0 Oct 30 02:13:31.375782 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:31.375577 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-817fd7-764dbbc7f9-9fsbh" Oct 30 02:13:31.375782 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:31.375619 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-817fd7-764dbbc7f9-9fsbh" event={"ID":"f8de244f-5cf0-4543-9118-6684c2be86f0","Type":"ContainerDied","Data":"73ad6962f718bda9b4dc6453a8ed6a38e66d6a98cd0e2813f05aae1443cf9f12"} Oct 30 02:13:31.375782 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:31.375677 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-817fd7-764dbbc7f9-9fsbh" event={"ID":"f8de244f-5cf0-4543-9118-6684c2be86f0","Type":"ContainerDied","Data":"180620a2f20cd31938232d1cdf2c6db1d14aa4e6c72c3d0fdd1290bfd5ac68cd"} Oct 30 02:13:31.375782 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:31.375708 6221 scope.go:122] "RemoveContainer" containerID="73ad6962f718bda9b4dc6453a8ed6a38e66d6a98cd0e2813f05aae1443cf9f12" Oct 30 02:13:31.386890 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:31.386861 6221 scope.go:122] "RemoveContainer" containerID="73ad6962f718bda9b4dc6453a8ed6a38e66d6a98cd0e2813f05aae1443cf9f12" Oct 30 02:13:31.387165 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:13:31.387146 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73ad6962f718bda9b4dc6453a8ed6a38e66d6a98cd0e2813f05aae1443cf9f12\": container with ID starting with 73ad6962f718bda9b4dc6453a8ed6a38e66d6a98cd0e2813f05aae1443cf9f12 not found: ID does not exist" containerID="73ad6962f718bda9b4dc6453a8ed6a38e66d6a98cd0e2813f05aae1443cf9f12" Oct 30 02:13:31.387236 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:31.387172 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73ad6962f718bda9b4dc6453a8ed6a38e66d6a98cd0e2813f05aae1443cf9f12"} err="failed to get container status \"73ad6962f718bda9b4dc6453a8ed6a38e66d6a98cd0e2813f05aae1443cf9f12\": rpc error: code = NotFound desc = could not find container \"73ad6962f718bda9b4dc6453a8ed6a38e66d6a98cd0e2813f05aae1443cf9f12\": container with ID starting with 73ad6962f718bda9b4dc6453a8ed6a38e66d6a98cd0e2813f05aae1443cf9f12 not found: ID does not exist" Oct 30 02:13:31.430376 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:31.430294 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-817fd7-764dbbc7f9-9fsbh"] Oct 30 02:13:31.442609 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:31.442582 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-817fd7-764dbbc7f9-9fsbh"] Oct 30 02:13:32.548896 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:32.548864 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8de244f-5cf0-4543-9118-6684c2be86f0" path="/var/lib/kubelet/pods/f8de244f-5cf0-4543-9118-6684c2be86f0/volumes" Oct 30 02:13:37.195904 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:37.195863 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-f81908-55df8db8bb-dlxz9" podUID="3af28e5b-f8a6-46b4-af05-f8ea92975c01" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:13:39.362764 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:39.362677 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2d5aa9-698b999554-z8f24" Oct 30 02:13:44.242813 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:44.242775 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k"] Oct 30 02:13:44.243655 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:44.243636 6221 cpu_manager.go:414] "containerMap: removing container" podUID="f8de244f-5cf0-4543-9118-6684c2be86f0" containerName="maas-api" Oct 30 02:13:44.243706 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:44.243658 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8de244f-5cf0-4543-9118-6684c2be86f0" containerName="maas-api" Oct 30 02:13:44.243800 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:44.243791 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8de244f-5cf0-4543-9118-6684c2be86f0" containerName="maas-api" Oct 30 02:13:44.248946 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:44.248928 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k" Oct 30 02:13:44.265478 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:44.265453 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k"] Oct 30 02:13:44.286313 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:44.286279 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-f89697-openshift-default-dockercfg-6j4rg\"" Oct 30 02:13:44.343012 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:44.342975 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc9f5\" (UniqueName: \"kubernetes.io/projected/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-kube-api-access-mc9f5\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k\" (UID: \"ff0dc863-4ea6-454f-9f61-5bfd9a9db849\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k" Oct 30 02:13:44.343271 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:44.343249 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-istio-podinfo\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k\" (UID: \"ff0dc863-4ea6-454f-9f61-5bfd9a9db849\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k" Oct 30 02:13:44.343478 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:44.343442 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-credential-socket\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k\" (UID: \"ff0dc863-4ea6-454f-9f61-5bfd9a9db849\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k" Oct 30 02:13:44.343675 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:44.343635 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-istiod-ca-cert\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k\" (UID: \"ff0dc863-4ea6-454f-9f61-5bfd9a9db849\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k" Oct 30 02:13:44.344018 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:44.343993 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-istio-envoy\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k\" (UID: \"ff0dc863-4ea6-454f-9f61-5bfd9a9db849\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k" Oct 30 02:13:44.344178 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:44.344160 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-istio-token\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k\" (UID: \"ff0dc863-4ea6-454f-9f61-5bfd9a9db849\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k" Oct 30 02:13:44.344404 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:44.344385 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-workload-certs\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k\" (UID: \"ff0dc863-4ea6-454f-9f61-5bfd9a9db849\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k" Oct 30 02:13:44.344612 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:44.344587 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-workload-socket\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k\" (UID: \"ff0dc863-4ea6-454f-9f61-5bfd9a9db849\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k" Oct 30 02:13:44.344717 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:44.344639 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-istio-data\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k\" (UID: \"ff0dc863-4ea6-454f-9f61-5bfd9a9db849\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k" Oct 30 02:13:44.446398 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:44.446359 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-istio-data\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k\" (UID: \"ff0dc863-4ea6-454f-9f61-5bfd9a9db849\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k" Oct 30 02:13:44.446595 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:44.446473 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc9f5\" (UniqueName: \"kubernetes.io/projected/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-kube-api-access-mc9f5\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k\" (UID: \"ff0dc863-4ea6-454f-9f61-5bfd9a9db849\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k" Oct 30 02:13:44.446595 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:44.446543 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-istio-podinfo\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k\" (UID: \"ff0dc863-4ea6-454f-9f61-5bfd9a9db849\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k" Oct 30 02:13:44.446724 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:44.446644 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-credential-socket\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k\" (UID: \"ff0dc863-4ea6-454f-9f61-5bfd9a9db849\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k" Oct 30 02:13:44.446724 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:44.446686 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-istiod-ca-cert\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k\" (UID: \"ff0dc863-4ea6-454f-9f61-5bfd9a9db849\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k" Oct 30 02:13:44.446837 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:44.446746 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-istio-data\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k\" (UID: \"ff0dc863-4ea6-454f-9f61-5bfd9a9db849\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k" Oct 30 02:13:44.446837 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:44.446780 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-istio-envoy\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k\" (UID: \"ff0dc863-4ea6-454f-9f61-5bfd9a9db849\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k" Oct 30 02:13:44.446837 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:44.446804 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-istio-token\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k\" (UID: \"ff0dc863-4ea6-454f-9f61-5bfd9a9db849\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k" Oct 30 02:13:44.447047 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:44.446871 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-workload-certs\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k\" (UID: \"ff0dc863-4ea6-454f-9f61-5bfd9a9db849\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k" Oct 30 02:13:44.447047 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:44.446905 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-workload-socket\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k\" (UID: \"ff0dc863-4ea6-454f-9f61-5bfd9a9db849\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k" Oct 30 02:13:44.447300 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:44.447276 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-workload-socket\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k\" (UID: \"ff0dc863-4ea6-454f-9f61-5bfd9a9db849\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k" Oct 30 02:13:44.447521 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:44.447499 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-workload-certs\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k\" (UID: \"ff0dc863-4ea6-454f-9f61-5bfd9a9db849\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k" Oct 30 02:13:44.447616 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:44.447557 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-istiod-ca-cert\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k\" (UID: \"ff0dc863-4ea6-454f-9f61-5bfd9a9db849\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k" Oct 30 02:13:44.447887 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:44.447868 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-credential-socket\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k\" (UID: \"ff0dc863-4ea6-454f-9f61-5bfd9a9db849\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k" Oct 30 02:13:44.449344 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:44.449321 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-istio-envoy\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k\" (UID: \"ff0dc863-4ea6-454f-9f61-5bfd9a9db849\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k" Oct 30 02:13:44.449594 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:44.449572 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-istio-podinfo\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k\" (UID: \"ff0dc863-4ea6-454f-9f61-5bfd9a9db849\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k" Oct 30 02:13:44.468463 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:44.468433 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc9f5\" (UniqueName: \"kubernetes.io/projected/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-kube-api-access-mc9f5\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k\" (UID: \"ff0dc863-4ea6-454f-9f61-5bfd9a9db849\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k" Oct 30 02:13:44.468793 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:44.468769 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-istio-token\") pod \"e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k\" (UID: \"ff0dc863-4ea6-454f-9f61-5bfd9a9db849\") " pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k" Oct 30 02:13:44.565174 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:44.565141 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k" Oct 30 02:13:44.886796 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:44.886768 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k"] Oct 30 02:13:44.888058 ip-10-0-135-65 kubenswrapper[6221]: W1030 02:13:44.888029 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff0dc863_4ea6_454f_9f61_5bfd9a9db849.slice/crio-30577fbf9c731884e7e42e83bc9072a62094e348057c5a926acc2c5311c1ce74 WatchSource:0}: Error finding container 30577fbf9c731884e7e42e83bc9072a62094e348057c5a926acc2c5311c1ce74: Status 404 returned error can't find the container with id 30577fbf9c731884e7e42e83bc9072a62094e348057c5a926acc2c5311c1ce74 Oct 30 02:13:44.891084 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:44.891048 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:13:44.891193 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:44.891133 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:13:44.891193 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:44.891179 6221 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 30 02:13:45.511771 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:45.511733 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k" event={"ID":"ff0dc863-4ea6-454f-9f61-5bfd9a9db849","Type":"ContainerStarted","Data":"bf266f7b670d5a0abe14a1284abce009f29ad5cce80a62ea132ee6c59e9526a4"} Oct 30 02:13:45.512185 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:45.511791 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k" event={"ID":"ff0dc863-4ea6-454f-9f61-5bfd9a9db849","Type":"ContainerStarted","Data":"30577fbf9c731884e7e42e83bc9072a62094e348057c5a926acc2c5311c1ce74"} Oct 30 02:13:45.565409 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:45.565374 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k" Oct 30 02:13:45.567681 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:45.567652 6221 patch_prober.go:29] interesting pod/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Oct 30 02:13:45.567838 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:45.567706 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k" podUID="ff0dc863-4ea6-454f-9f61-5bfd9a9db849" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Oct 30 02:13:45.581168 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:45.581120 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k" podStartSLOduration=1.5811064369999999 podStartE2EDuration="1.581106437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:13:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:13:45.565801505 +0000 UTC m=+3349.144088843" watchObservedRunningTime="2026-10-30 02:13:45.581106437 +0000 UTC m=+3349.159393824" Oct 30 02:13:46.565992 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:46.565954 6221 patch_prober.go:29] interesting pod/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Oct 30 02:13:46.566426 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:46.566008 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k" podUID="ff0dc863-4ea6-454f-9f61-5bfd9a9db849" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Oct 30 02:13:47.354506 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:47.354473 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f81908-55df8db8bb-dlxz9" Oct 30 02:13:47.402518 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:47.402478 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3af28e5b-f8a6-46b4-af05-f8ea92975c01-kube-api-access-lfpqb\" (UniqueName: \"kubernetes.io/projected/3af28e5b-f8a6-46b4-af05-f8ea92975c01-kube-api-access-lfpqb\") pod \"3af28e5b-f8a6-46b4-af05-f8ea92975c01\" (UID: \"3af28e5b-f8a6-46b4-af05-f8ea92975c01\") " Oct 30 02:13:47.402734 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:47.402595 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3af28e5b-f8a6-46b4-af05-f8ea92975c01-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3af28e5b-f8a6-46b4-af05-f8ea92975c01-plugins-config-volume\") pod \"3af28e5b-f8a6-46b4-af05-f8ea92975c01\" (UID: \"3af28e5b-f8a6-46b4-af05-f8ea92975c01\") " Oct 30 02:13:47.403507 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:47.403474 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3af28e5b-f8a6-46b4-af05-f8ea92975c01-plugins-config-volume" pod "3af28e5b-f8a6-46b4-af05-f8ea92975c01" (UID: "3af28e5b-f8a6-46b4-af05-f8ea92975c01"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:13:47.439775 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:47.439736 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3af28e5b-f8a6-46b4-af05-f8ea92975c01-kube-api-access-lfpqb" pod "3af28e5b-f8a6-46b4-af05-f8ea92975c01" (UID: "3af28e5b-f8a6-46b4-af05-f8ea92975c01"). InnerVolumeSpecName "kube-api-access-lfpqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:13:47.505133 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:47.505094 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3af28e5b-f8a6-46b4-af05-f8ea92975c01-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:13:47.505133 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:47.505136 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfpqb\" (UniqueName: \"kubernetes.io/projected/3af28e5b-f8a6-46b4-af05-f8ea92975c01-kube-api-access-lfpqb\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:13:47.550899 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:47.550866 6221 generic.go:356] "Generic (PLEG): container finished" podID="3af28e5b-f8a6-46b4-af05-f8ea92975c01" containerID="824807c5eda587ce9b3b7a2c51fd761c2abacdce8cdc1230f74f98ce1aee6296" exitCode=0 Oct 30 02:13:47.551095 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:47.550914 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f81908-55df8db8bb-dlxz9" Oct 30 02:13:47.551095 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:47.550945 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f81908-55df8db8bb-dlxz9" event={"ID":"3af28e5b-f8a6-46b4-af05-f8ea92975c01","Type":"ContainerDied","Data":"824807c5eda587ce9b3b7a2c51fd761c2abacdce8cdc1230f74f98ce1aee6296"} Oct 30 02:13:47.551095 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:47.550992 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f81908-55df8db8bb-dlxz9" event={"ID":"3af28e5b-f8a6-46b4-af05-f8ea92975c01","Type":"ContainerDied","Data":"e16676b3b6f2a6b4360b226b9b523604c241b5b68da03a544151f575a5b59b94"} Oct 30 02:13:47.551095 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:47.551016 6221 scope.go:122] "RemoveContainer" containerID="824807c5eda587ce9b3b7a2c51fd761c2abacdce8cdc1230f74f98ce1aee6296" Oct 30 02:13:47.561897 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:47.561880 6221 scope.go:122] "RemoveContainer" containerID="824807c5eda587ce9b3b7a2c51fd761c2abacdce8cdc1230f74f98ce1aee6296" Oct 30 02:13:47.562161 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:13:47.562144 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"824807c5eda587ce9b3b7a2c51fd761c2abacdce8cdc1230f74f98ce1aee6296\": container with ID starting with 824807c5eda587ce9b3b7a2c51fd761c2abacdce8cdc1230f74f98ce1aee6296 not found: ID does not exist" containerID="824807c5eda587ce9b3b7a2c51fd761c2abacdce8cdc1230f74f98ce1aee6296" Oct 30 02:13:47.562202 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:47.562168 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"824807c5eda587ce9b3b7a2c51fd761c2abacdce8cdc1230f74f98ce1aee6296"} err="failed to get container status \"824807c5eda587ce9b3b7a2c51fd761c2abacdce8cdc1230f74f98ce1aee6296\": rpc error: code = NotFound desc = could not find container \"824807c5eda587ce9b3b7a2c51fd761c2abacdce8cdc1230f74f98ce1aee6296\": container with ID starting with 824807c5eda587ce9b3b7a2c51fd761c2abacdce8cdc1230f74f98ce1aee6296 not found: ID does not exist" Oct 30 02:13:47.565467 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:47.565448 6221 patch_prober.go:29] interesting pod/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Oct 30 02:13:47.565598 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:47.565507 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k" podUID="ff0dc863-4ea6-454f-9f61-5bfd9a9db849" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Oct 30 02:13:47.603393 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:47.603361 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f81908-55df8db8bb-dlxz9"] Oct 30 02:13:47.634285 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:47.634254 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f81908-55df8db8bb-dlxz9"] Oct 30 02:13:48.231861 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:48.231691 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-f81908-55df8db8bb-dlxz9" podUID="3af28e5b-f8a6-46b4-af05-f8ea92975c01" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.199:50052\" within 1s: context deadline exceeded" Oct 30 02:13:48.231861 ip-10-0-135-65 kubenswrapper[6221]: W1030 02:13:48.231798 6221 logging.go:55] [core] [Channel #14468 SubChannel #14469]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.199:50052", ServerName: "10.134.0.199:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.199:50052: operation was canceled" Oct 30 02:13:48.549903 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:48.549869 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3af28e5b-f8a6-46b4-af05-f8ea92975c01" path="/var/lib/kubelet/pods/3af28e5b-f8a6-46b4-af05-f8ea92975c01/volumes" Oct 30 02:13:48.566348 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:48.566317 6221 patch_prober.go:29] interesting pod/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Oct 30 02:13:48.566501 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:48.566367 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k" podUID="ff0dc863-4ea6-454f-9f61-5bfd9a9db849" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Oct 30 02:13:49.565525 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:49.565476 6221 patch_prober.go:29] interesting pod/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Oct 30 02:13:49.565965 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:49.565567 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k" podUID="ff0dc863-4ea6-454f-9f61-5bfd9a9db849" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Oct 30 02:13:50.566446 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:50.566411 6221 patch_prober.go:29] interesting pod/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Oct 30 02:13:50.566899 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:50.566466 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k" podUID="ff0dc863-4ea6-454f-9f61-5bfd9a9db849" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Oct 30 02:13:51.496849 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:13:51.496812 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 02:13:51.566312 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:51.566274 6221 patch_prober.go:29] interesting pod/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body= Oct 30 02:13:51.566524 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:51.566325 6221 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k" podUID="ff0dc863-4ea6-454f-9f61-5bfd9a9db849" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" Oct 30 02:13:52.570866 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:52.570836 6221 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k" Oct 30 02:13:52.571286 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:52.571271 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k" Oct 30 02:13:52.571871 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:13:52.571850 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k" Oct 30 02:14:01.638325 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:01.638280 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp"] Oct 30 02:14:01.639216 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:01.639165 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp" podUID="410ab4b3-ac20-4633-8311-763564d670e4" containerName="istio-proxy" containerID="cri-o://9b32bb2c137f4eca7b288669b28dd98133d4cd6c1b2062aca640f3fe4b1d5fbb" gracePeriod=30 Oct 30 02:14:03.347740 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:03.347699 6221 patch_prober.go:29] interesting pod/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:14:03.348204 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:03.347781 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp" podUID="410ab4b3-ac20-4633-8311-763564d670e4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:14:06.928703 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:06.928676 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp" Oct 30 02:14:07.037293 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:07.037253 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/410ab4b3-ac20-4633-8311-763564d670e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/410ab4b3-ac20-4633-8311-763564d670e4-credential-socket\") pod \"410ab4b3-ac20-4633-8311-763564d670e4\" (UID: \"410ab4b3-ac20-4633-8311-763564d670e4\") " Oct 30 02:14:07.037521 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:07.037339 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/410ab4b3-ac20-4633-8311-763564d670e4-istio-token\" (UniqueName: \"kubernetes.io/projected/410ab4b3-ac20-4633-8311-763564d670e4-istio-token\") pod \"410ab4b3-ac20-4633-8311-763564d670e4\" (UID: \"410ab4b3-ac20-4633-8311-763564d670e4\") " Oct 30 02:14:07.037521 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:07.037395 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/410ab4b3-ac20-4633-8311-763564d670e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/410ab4b3-ac20-4633-8311-763564d670e4-istio-data\") pod \"410ab4b3-ac20-4633-8311-763564d670e4\" (UID: \"410ab4b3-ac20-4633-8311-763564d670e4\") " Oct 30 02:14:07.037723 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:07.037525 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/410ab4b3-ac20-4633-8311-763564d670e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/410ab4b3-ac20-4633-8311-763564d670e4-workload-certs\") pod \"410ab4b3-ac20-4633-8311-763564d670e4\" (UID: \"410ab4b3-ac20-4633-8311-763564d670e4\") " Oct 30 02:14:07.037723 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:07.037576 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/410ab4b3-ac20-4633-8311-763564d670e4-credential-socket" pod "410ab4b3-ac20-4633-8311-763564d670e4" (UID: "410ab4b3-ac20-4633-8311-763564d670e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:07.037841 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:07.037758 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/410ab4b3-ac20-4633-8311-763564d670e4-workload-certs" pod "410ab4b3-ac20-4633-8311-763564d670e4" (UID: "410ab4b3-ac20-4633-8311-763564d670e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:07.037841 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:07.037768 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/410ab4b3-ac20-4633-8311-763564d670e4-istio-data" pod "410ab4b3-ac20-4633-8311-763564d670e4" (UID: "410ab4b3-ac20-4633-8311-763564d670e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:07.037841 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:07.037775 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/410ab4b3-ac20-4633-8311-763564d670e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/410ab4b3-ac20-4633-8311-763564d670e4-istio-envoy\") pod \"410ab4b3-ac20-4633-8311-763564d670e4\" (UID: \"410ab4b3-ac20-4633-8311-763564d670e4\") " Oct 30 02:14:07.038027 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:07.037867 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/410ab4b3-ac20-4633-8311-763564d670e4-kube-api-access-l9pxk\" (UniqueName: \"kubernetes.io/projected/410ab4b3-ac20-4633-8311-763564d670e4-kube-api-access-l9pxk\") pod \"410ab4b3-ac20-4633-8311-763564d670e4\" (UID: \"410ab4b3-ac20-4633-8311-763564d670e4\") " Oct 30 02:14:07.038027 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:07.037924 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/410ab4b3-ac20-4633-8311-763564d670e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/410ab4b3-ac20-4633-8311-763564d670e4-istiod-ca-cert\") pod \"410ab4b3-ac20-4633-8311-763564d670e4\" (UID: \"410ab4b3-ac20-4633-8311-763564d670e4\") " Oct 30 02:14:07.038027 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:07.037974 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/410ab4b3-ac20-4633-8311-763564d670e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/410ab4b3-ac20-4633-8311-763564d670e4-istio-podinfo\") pod \"410ab4b3-ac20-4633-8311-763564d670e4\" (UID: \"410ab4b3-ac20-4633-8311-763564d670e4\") " Oct 30 02:14:07.038200 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:07.038082 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/410ab4b3-ac20-4633-8311-763564d670e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/410ab4b3-ac20-4633-8311-763564d670e4-workload-socket\") pod \"410ab4b3-ac20-4633-8311-763564d670e4\" (UID: \"410ab4b3-ac20-4633-8311-763564d670e4\") " Oct 30 02:14:07.038289 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:07.038265 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/410ab4b3-ac20-4633-8311-763564d670e4-istiod-ca-cert" pod "410ab4b3-ac20-4633-8311-763564d670e4" (UID: "410ab4b3-ac20-4633-8311-763564d670e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:14:07.038365 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:07.038335 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/410ab4b3-ac20-4633-8311-763564d670e4-workload-socket" pod "410ab4b3-ac20-4633-8311-763564d670e4" (UID: "410ab4b3-ac20-4633-8311-763564d670e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:07.039191 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:07.039162 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/410ab4b3-ac20-4633-8311-763564d670e4-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:14:07.039191 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:07.039193 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/410ab4b3-ac20-4633-8311-763564d670e4-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:14:07.039383 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:07.039207 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/410ab4b3-ac20-4633-8311-763564d670e4-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:14:07.039383 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:07.039220 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/410ab4b3-ac20-4633-8311-763564d670e4-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:14:07.039383 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:07.039233 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/410ab4b3-ac20-4633-8311-763564d670e4-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:14:07.040223 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:07.040174 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/410ab4b3-ac20-4633-8311-763564d670e4-istio-envoy" pod "410ab4b3-ac20-4633-8311-763564d670e4" (UID: "410ab4b3-ac20-4633-8311-763564d670e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:07.040578 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:07.040524 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/410ab4b3-ac20-4633-8311-763564d670e4-istio-token" pod "410ab4b3-ac20-4633-8311-763564d670e4" (UID: "410ab4b3-ac20-4633-8311-763564d670e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:14:07.040798 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:07.040767 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/410ab4b3-ac20-4633-8311-763564d670e4-kube-api-access-l9pxk" pod "410ab4b3-ac20-4633-8311-763564d670e4" (UID: "410ab4b3-ac20-4633-8311-763564d670e4"). InnerVolumeSpecName "kube-api-access-l9pxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:14:07.041199 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:07.041178 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/410ab4b3-ac20-4633-8311-763564d670e4-istio-podinfo" pod "410ab4b3-ac20-4633-8311-763564d670e4" (UID: "410ab4b3-ac20-4633-8311-763564d670e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:14:07.144156 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:07.141340 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/410ab4b3-ac20-4633-8311-763564d670e4-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:14:07.144156 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:07.141383 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9pxk\" (UniqueName: \"kubernetes.io/projected/410ab4b3-ac20-4633-8311-763564d670e4-kube-api-access-l9pxk\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:14:07.144156 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:07.141402 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/410ab4b3-ac20-4633-8311-763564d670e4-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:14:07.144156 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:07.141415 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/410ab4b3-ac20-4633-8311-763564d670e4-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:14:07.712975 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:07.712937 6221 generic.go:356] "Generic (PLEG): container finished" podID="410ab4b3-ac20-4633-8311-763564d670e4" containerID="9b32bb2c137f4eca7b288669b28dd98133d4cd6c1b2062aca640f3fe4b1d5fbb" exitCode=0 Oct 30 02:14:07.713201 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:07.712993 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp" Oct 30 02:14:07.713201 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:07.713022 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp" event={"ID":"410ab4b3-ac20-4633-8311-763564d670e4","Type":"ContainerDied","Data":"9b32bb2c137f4eca7b288669b28dd98133d4cd6c1b2062aca640f3fe4b1d5fbb"} Oct 30 02:14:07.713201 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:07.713078 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp" event={"ID":"410ab4b3-ac20-4633-8311-763564d670e4","Type":"ContainerDied","Data":"14c49e7c096217ca3f19939973775bfb9b030d1c9dd4128bfd45877c8c02cc0f"} Oct 30 02:14:07.713201 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:07.713101 6221 scope.go:122] "RemoveContainer" containerID="9b32bb2c137f4eca7b288669b28dd98133d4cd6c1b2062aca640f3fe4b1d5fbb" Oct 30 02:14:07.725054 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:07.725036 6221 scope.go:122] "RemoveContainer" containerID="9b32bb2c137f4eca7b288669b28dd98133d4cd6c1b2062aca640f3fe4b1d5fbb" Oct 30 02:14:07.725351 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:14:07.725331 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b32bb2c137f4eca7b288669b28dd98133d4cd6c1b2062aca640f3fe4b1d5fbb\": container with ID starting with 9b32bb2c137f4eca7b288669b28dd98133d4cd6c1b2062aca640f3fe4b1d5fbb not found: ID does not exist" containerID="9b32bb2c137f4eca7b288669b28dd98133d4cd6c1b2062aca640f3fe4b1d5fbb" Oct 30 02:14:07.725401 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:07.725359 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b32bb2c137f4eca7b288669b28dd98133d4cd6c1b2062aca640f3fe4b1d5fbb"} err="failed to get container status \"9b32bb2c137f4eca7b288669b28dd98133d4cd6c1b2062aca640f3fe4b1d5fbb\": rpc error: code = NotFound desc = could not find container \"9b32bb2c137f4eca7b288669b28dd98133d4cd6c1b2062aca640f3fe4b1d5fbb\": container with ID starting with 9b32bb2c137f4eca7b288669b28dd98133d4cd6c1b2062aca640f3fe4b1d5fbb not found: ID does not exist" Oct 30 02:14:07.779299 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:07.779262 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp"] Oct 30 02:14:07.783464 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:07.783431 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-803472-openshift-default-c97844bc7-jbgtp"] Oct 30 02:14:08.549547 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:08.549498 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="410ab4b3-ac20-4633-8311-763564d670e4" path="/var/lib/kubelet/pods/410ab4b3-ac20-4633-8311-763564d670e4/volumes" Oct 30 02:14:08.748580 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:08.748549 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k"] Oct 30 02:14:08.748841 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:08.748800 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k" podUID="066ee435-d202-448a-ad09-17142e92ae5f" containerName="istio-proxy" containerID="cri-o://395646c3dfd949809591a7ab2c58cf8f02bead0e35512c49410cdc0ec03b110b" gracePeriod=30 Oct 30 02:14:13.389184 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:13.389150 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2d5aa9-856559788f-hv2fv"] Oct 30 02:14:13.390048 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:13.390028 6221 cpu_manager.go:414] "containerMap: removing container" podUID="410ab4b3-ac20-4633-8311-763564d670e4" containerName="istio-proxy" Oct 30 02:14:13.390048 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:13.390049 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="410ab4b3-ac20-4633-8311-763564d670e4" containerName="istio-proxy" Oct 30 02:14:13.390189 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:13.390111 6221 cpu_manager.go:414] "containerMap: removing container" podUID="3af28e5b-f8a6-46b4-af05-f8ea92975c01" containerName="payload-processing" Oct 30 02:14:13.390189 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:13.390117 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3af28e5b-f8a6-46b4-af05-f8ea92975c01" containerName="payload-processing" Oct 30 02:14:13.390262 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:13.390223 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="3af28e5b-f8a6-46b4-af05-f8ea92975c01" containerName="payload-processing" Oct 30 02:14:13.390262 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:13.390235 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="410ab4b3-ac20-4633-8311-763564d670e4" containerName="istio-proxy" Oct 30 02:14:13.396040 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:13.396021 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2d5aa9-856559788f-hv2fv" Oct 30 02:14:13.413971 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:13.413940 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2d5aa9-856559788f-hv2fv"] Oct 30 02:14:13.420951 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:13.420924 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-2d5aa9-serving-cert\"" Oct 30 02:14:13.483822 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:13.483787 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d7tl\" (UniqueName: \"kubernetes.io/projected/fbab4591-a274-4c63-9004-8c05dd8676df-kube-api-access-9d7tl\") pod \"maas-api-e2e-inf-b-2d5aa9-856559788f-hv2fv\" (UID: \"fbab4591-a274-4c63-9004-8c05dd8676df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2d5aa9-856559788f-hv2fv" Oct 30 02:14:13.483995 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:13.483853 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbab4591-a274-4c63-9004-8c05dd8676df-metrics-certs\") pod \"maas-api-e2e-inf-b-2d5aa9-856559788f-hv2fv\" (UID: \"fbab4591-a274-4c63-9004-8c05dd8676df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2d5aa9-856559788f-hv2fv" Oct 30 02:14:13.484040 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:13.483980 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbab4591-a274-4c63-9004-8c05dd8676df-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2d5aa9-856559788f-hv2fv\" (UID: \"fbab4591-a274-4c63-9004-8c05dd8676df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2d5aa9-856559788f-hv2fv" Oct 30 02:14:13.484117 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:13.484098 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbab4591-a274-4c63-9004-8c05dd8676df-maas-api-tls\") pod \"maas-api-e2e-inf-b-2d5aa9-856559788f-hv2fv\" (UID: \"fbab4591-a274-4c63-9004-8c05dd8676df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2d5aa9-856559788f-hv2fv" Oct 30 02:14:13.585289 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:13.585241 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d7tl\" (UniqueName: \"kubernetes.io/projected/fbab4591-a274-4c63-9004-8c05dd8676df-kube-api-access-9d7tl\") pod \"maas-api-e2e-inf-b-2d5aa9-856559788f-hv2fv\" (UID: \"fbab4591-a274-4c63-9004-8c05dd8676df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2d5aa9-856559788f-hv2fv" Oct 30 02:14:13.585564 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:13.585314 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbab4591-a274-4c63-9004-8c05dd8676df-metrics-certs\") pod \"maas-api-e2e-inf-b-2d5aa9-856559788f-hv2fv\" (UID: \"fbab4591-a274-4c63-9004-8c05dd8676df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2d5aa9-856559788f-hv2fv" Oct 30 02:14:13.585564 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:13.585350 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbab4591-a274-4c63-9004-8c05dd8676df-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2d5aa9-856559788f-hv2fv\" (UID: \"fbab4591-a274-4c63-9004-8c05dd8676df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2d5aa9-856559788f-hv2fv" Oct 30 02:14:13.585564 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:13.585395 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbab4591-a274-4c63-9004-8c05dd8676df-maas-api-tls\") pod \"maas-api-e2e-inf-b-2d5aa9-856559788f-hv2fv\" (UID: \"fbab4591-a274-4c63-9004-8c05dd8676df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2d5aa9-856559788f-hv2fv" Oct 30 02:14:13.586129 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:13.586102 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbab4591-a274-4c63-9004-8c05dd8676df-openshift-service-ca\") pod \"maas-api-e2e-inf-b-2d5aa9-856559788f-hv2fv\" (UID: \"fbab4591-a274-4c63-9004-8c05dd8676df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2d5aa9-856559788f-hv2fv" Oct 30 02:14:13.587921 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:13.587896 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbab4591-a274-4c63-9004-8c05dd8676df-maas-api-tls\") pod \"maas-api-e2e-inf-b-2d5aa9-856559788f-hv2fv\" (UID: \"fbab4591-a274-4c63-9004-8c05dd8676df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2d5aa9-856559788f-hv2fv" Oct 30 02:14:13.588009 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:13.587922 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbab4591-a274-4c63-9004-8c05dd8676df-metrics-certs\") pod \"maas-api-e2e-inf-b-2d5aa9-856559788f-hv2fv\" (UID: \"fbab4591-a274-4c63-9004-8c05dd8676df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2d5aa9-856559788f-hv2fv" Oct 30 02:14:13.640614 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:13.638065 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d7tl\" (UniqueName: \"kubernetes.io/projected/fbab4591-a274-4c63-9004-8c05dd8676df-kube-api-access-9d7tl\") pod \"maas-api-e2e-inf-b-2d5aa9-856559788f-hv2fv\" (UID: \"fbab4591-a274-4c63-9004-8c05dd8676df\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2d5aa9-856559788f-hv2fv" Oct 30 02:14:13.710680 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:13.710617 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2d5aa9-856559788f-hv2fv" Oct 30 02:14:14.012829 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:14.012803 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k" Oct 30 02:14:14.024813 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:14.024790 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2d5aa9-856559788f-hv2fv"] Oct 30 02:14:14.027085 ip-10-0-135-65 kubenswrapper[6221]: W1030 02:14:14.027060 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbab4591_a274_4c63_9004_8c05dd8676df.slice/crio-4aad5ffac3e294a7a136a10c33c71bb72c4efd9ea79fea3d32d095499863f539 WatchSource:0}: Error finding container 4aad5ffac3e294a7a136a10c33c71bb72c4efd9ea79fea3d32d095499863f539: Status 404 returned error can't find the container with id 4aad5ffac3e294a7a136a10c33c71bb72c4efd9ea79fea3d32d095499863f539 Oct 30 02:14:14.028624 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:14.028603 6221 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 02:14:14.103481 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:14.103438 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/066ee435-d202-448a-ad09-17142e92ae5f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/066ee435-d202-448a-ad09-17142e92ae5f-istio-data\") pod \"066ee435-d202-448a-ad09-17142e92ae5f\" (UID: \"066ee435-d202-448a-ad09-17142e92ae5f\") " Oct 30 02:14:14.103704 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:14.103665 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/066ee435-d202-448a-ad09-17142e92ae5f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/066ee435-d202-448a-ad09-17142e92ae5f-workload-socket\") pod \"066ee435-d202-448a-ad09-17142e92ae5f\" (UID: \"066ee435-d202-448a-ad09-17142e92ae5f\") " Oct 30 02:14:14.103787 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:14.103715 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/066ee435-d202-448a-ad09-17142e92ae5f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/066ee435-d202-448a-ad09-17142e92ae5f-istio-podinfo\") pod \"066ee435-d202-448a-ad09-17142e92ae5f\" (UID: \"066ee435-d202-448a-ad09-17142e92ae5f\") " Oct 30 02:14:14.103787 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:14.103718 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/066ee435-d202-448a-ad09-17142e92ae5f-istio-data" pod "066ee435-d202-448a-ad09-17142e92ae5f" (UID: "066ee435-d202-448a-ad09-17142e92ae5f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:14.103906 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:14.103854 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/066ee435-d202-448a-ad09-17142e92ae5f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/066ee435-d202-448a-ad09-17142e92ae5f-workload-certs\") pod \"066ee435-d202-448a-ad09-17142e92ae5f\" (UID: \"066ee435-d202-448a-ad09-17142e92ae5f\") " Oct 30 02:14:14.103963 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:14.103909 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/066ee435-d202-448a-ad09-17142e92ae5f-istio-token\" (UniqueName: \"kubernetes.io/projected/066ee435-d202-448a-ad09-17142e92ae5f-istio-token\") pod \"066ee435-d202-448a-ad09-17142e92ae5f\" (UID: \"066ee435-d202-448a-ad09-17142e92ae5f\") " Oct 30 02:14:14.104031 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:14.103957 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/066ee435-d202-448a-ad09-17142e92ae5f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/066ee435-d202-448a-ad09-17142e92ae5f-istiod-ca-cert\") pod \"066ee435-d202-448a-ad09-17142e92ae5f\" (UID: \"066ee435-d202-448a-ad09-17142e92ae5f\") " Oct 30 02:14:14.104031 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:14.103960 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/066ee435-d202-448a-ad09-17142e92ae5f-workload-socket" pod "066ee435-d202-448a-ad09-17142e92ae5f" (UID: "066ee435-d202-448a-ad09-17142e92ae5f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:14.104031 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:14.103997 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/066ee435-d202-448a-ad09-17142e92ae5f-kube-api-access-m8wdn\" (UniqueName: \"kubernetes.io/projected/066ee435-d202-448a-ad09-17142e92ae5f-kube-api-access-m8wdn\") pod \"066ee435-d202-448a-ad09-17142e92ae5f\" (UID: \"066ee435-d202-448a-ad09-17142e92ae5f\") " Oct 30 02:14:14.104200 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:14.104040 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/066ee435-d202-448a-ad09-17142e92ae5f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/066ee435-d202-448a-ad09-17142e92ae5f-credential-socket\") pod \"066ee435-d202-448a-ad09-17142e92ae5f\" (UID: \"066ee435-d202-448a-ad09-17142e92ae5f\") " Oct 30 02:14:14.104200 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:14.104149 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/066ee435-d202-448a-ad09-17142e92ae5f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/066ee435-d202-448a-ad09-17142e92ae5f-istio-envoy\") pod \"066ee435-d202-448a-ad09-17142e92ae5f\" (UID: \"066ee435-d202-448a-ad09-17142e92ae5f\") " Oct 30 02:14:14.104435 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:14.104408 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/066ee435-d202-448a-ad09-17142e92ae5f-credential-socket" pod "066ee435-d202-448a-ad09-17142e92ae5f" (UID: "066ee435-d202-448a-ad09-17142e92ae5f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:14.104574 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:14.104549 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/066ee435-d202-448a-ad09-17142e92ae5f-istiod-ca-cert" pod "066ee435-d202-448a-ad09-17142e92ae5f" (UID: "066ee435-d202-448a-ad09-17142e92ae5f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:14:14.104896 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:14.104871 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/066ee435-d202-448a-ad09-17142e92ae5f-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:14:14.105013 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:14.104899 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/066ee435-d202-448a-ad09-17142e92ae5f-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:14:14.105013 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:14.104916 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/066ee435-d202-448a-ad09-17142e92ae5f-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:14:14.105013 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:14.104929 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/066ee435-d202-448a-ad09-17142e92ae5f-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:14:14.105408 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:14.105384 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/066ee435-d202-448a-ad09-17142e92ae5f-workload-certs" pod "066ee435-d202-448a-ad09-17142e92ae5f" (UID: "066ee435-d202-448a-ad09-17142e92ae5f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:14.106718 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:14.106681 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/066ee435-d202-448a-ad09-17142e92ae5f-istio-token" pod "066ee435-d202-448a-ad09-17142e92ae5f" (UID: "066ee435-d202-448a-ad09-17142e92ae5f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:14:14.106811 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:14.106774 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/066ee435-d202-448a-ad09-17142e92ae5f-istio-podinfo" pod "066ee435-d202-448a-ad09-17142e92ae5f" (UID: "066ee435-d202-448a-ad09-17142e92ae5f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:14:14.106857 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:14.106812 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/066ee435-d202-448a-ad09-17142e92ae5f-istio-envoy" pod "066ee435-d202-448a-ad09-17142e92ae5f" (UID: "066ee435-d202-448a-ad09-17142e92ae5f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:14:14.107600 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:14.107572 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/066ee435-d202-448a-ad09-17142e92ae5f-kube-api-access-m8wdn" pod "066ee435-d202-448a-ad09-17142e92ae5f" (UID: "066ee435-d202-448a-ad09-17142e92ae5f"). InnerVolumeSpecName "kube-api-access-m8wdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:14:14.206483 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:14.206398 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/066ee435-d202-448a-ad09-17142e92ae5f-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:14:14.206483 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:14.206430 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/066ee435-d202-448a-ad09-17142e92ae5f-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:14:14.206483 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:14.206440 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/066ee435-d202-448a-ad09-17142e92ae5f-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:14:14.206483 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:14.206449 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/066ee435-d202-448a-ad09-17142e92ae5f-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:14:14.206483 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:14.206458 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8wdn\" (UniqueName: \"kubernetes.io/projected/066ee435-d202-448a-ad09-17142e92ae5f-kube-api-access-m8wdn\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:14:14.779814 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:14.779712 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2d5aa9-856559788f-hv2fv" event={"ID":"fbab4591-a274-4c63-9004-8c05dd8676df","Type":"ContainerStarted","Data":"54633b5ade30d3be7aa012c92e47606beaccb118a05577ce67bd715d4dfa4cd9"} Oct 30 02:14:14.779814 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:14.779774 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2d5aa9-856559788f-hv2fv" event={"ID":"fbab4591-a274-4c63-9004-8c05dd8676df","Type":"ContainerStarted","Data":"4aad5ffac3e294a7a136a10c33c71bb72c4efd9ea79fea3d32d095499863f539"} Oct 30 02:14:14.779814 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:14.779803 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2d5aa9-856559788f-hv2fv" Oct 30 02:14:14.781053 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:14.781024 6221 generic.go:356] "Generic (PLEG): container finished" podID="066ee435-d202-448a-ad09-17142e92ae5f" containerID="395646c3dfd949809591a7ab2c58cf8f02bead0e35512c49410cdc0ec03b110b" exitCode=0 Oct 30 02:14:14.781166 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:14.781065 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k" Oct 30 02:14:14.781166 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:14.781113 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k" event={"ID":"066ee435-d202-448a-ad09-17142e92ae5f","Type":"ContainerDied","Data":"395646c3dfd949809591a7ab2c58cf8f02bead0e35512c49410cdc0ec03b110b"} Oct 30 02:14:14.781166 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:14.781162 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k" event={"ID":"066ee435-d202-448a-ad09-17142e92ae5f","Type":"ContainerDied","Data":"8f1d5cc85ee9c8b47ab994945db34470ccbdc3f4833feb634f5e754e25c5a90c"} Oct 30 02:14:14.781272 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:14.781190 6221 scope.go:122] "RemoveContainer" containerID="395646c3dfd949809591a7ab2c58cf8f02bead0e35512c49410cdc0ec03b110b" Oct 30 02:14:14.792920 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:14.792899 6221 scope.go:122] "RemoveContainer" containerID="395646c3dfd949809591a7ab2c58cf8f02bead0e35512c49410cdc0ec03b110b" Oct 30 02:14:14.793222 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:14:14.793203 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"395646c3dfd949809591a7ab2c58cf8f02bead0e35512c49410cdc0ec03b110b\": container with ID starting with 395646c3dfd949809591a7ab2c58cf8f02bead0e35512c49410cdc0ec03b110b not found: ID does not exist" containerID="395646c3dfd949809591a7ab2c58cf8f02bead0e35512c49410cdc0ec03b110b" Oct 30 02:14:14.793284 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:14.793231 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"395646c3dfd949809591a7ab2c58cf8f02bead0e35512c49410cdc0ec03b110b"} err="failed to get container status \"395646c3dfd949809591a7ab2c58cf8f02bead0e35512c49410cdc0ec03b110b\": rpc error: code = NotFound desc = could not find container \"395646c3dfd949809591a7ab2c58cf8f02bead0e35512c49410cdc0ec03b110b\": container with ID starting with 395646c3dfd949809591a7ab2c58cf8f02bead0e35512c49410cdc0ec03b110b not found: ID does not exist" Oct 30 02:14:14.823440 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:14.823384 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2d5aa9-856559788f-hv2fv" podStartSLOduration=1.444551058 podStartE2EDuration="1.823369197s" totalImagesPullingTime="378.818139ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:14:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:14:14.820013344 +0000 UTC m=+3378.398300680" watchObservedRunningTime="2026-10-30 02:14:14.823369197 +0000 UTC m=+3378.401656556" Oct 30 02:14:14.859791 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:14.859760 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k"] Oct 30 02:14:14.865207 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:14.865176 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f81908-openshift-default-8ddd46997-vh64k"] Oct 30 02:14:16.550551 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:16.550501 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="066ee435-d202-448a-ad09-17142e92ae5f" path="/var/lib/kubelet/pods/066ee435-d202-448a-ad09-17142e92ae5f/volumes" Oct 30 02:14:19.783708 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:19.783676 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2eae39-d8f4c6966-szxzt"] Oct 30 02:14:20.795397 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:20.795364 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2d5aa9-856559788f-hv2fv" Oct 30 02:14:21.494825 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:14:21.494789 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 02:14:24.785008 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:24.784945 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-2eae39-d8f4c6966-szxzt" podUID="b683ef8c-fc50-4c69-98cf-fc3a1f59d6c1" containerName="payload-processing" containerID="cri-o://229e01cc8e36b137a5f9ff7d7d334862cbc6cfdd2db185e0d209f7cc71222c99" gracePeriod=25 Oct 30 02:14:25.061000 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:25.060971 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2eae39-d8f4c6966-szxzt" Oct 30 02:14:25.126346 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:25.126315 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b683ef8c-fc50-4c69-98cf-fc3a1f59d6c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b683ef8c-fc50-4c69-98cf-fc3a1f59d6c1-plugins-config-volume\") pod \"b683ef8c-fc50-4c69-98cf-fc3a1f59d6c1\" (UID: \"b683ef8c-fc50-4c69-98cf-fc3a1f59d6c1\") " Oct 30 02:14:25.126578 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:25.126526 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b683ef8c-fc50-4c69-98cf-fc3a1f59d6c1-kube-api-access-d9vxm\" (UniqueName: \"kubernetes.io/projected/b683ef8c-fc50-4c69-98cf-fc3a1f59d6c1-kube-api-access-d9vxm\") pod \"b683ef8c-fc50-4c69-98cf-fc3a1f59d6c1\" (UID: \"b683ef8c-fc50-4c69-98cf-fc3a1f59d6c1\") " Oct 30 02:14:25.126851 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:25.126820 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b683ef8c-fc50-4c69-98cf-fc3a1f59d6c1-plugins-config-volume" pod "b683ef8c-fc50-4c69-98cf-fc3a1f59d6c1" (UID: "b683ef8c-fc50-4c69-98cf-fc3a1f59d6c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:14:25.127192 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:25.127171 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b683ef8c-fc50-4c69-98cf-fc3a1f59d6c1-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:14:25.128791 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:25.128765 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b683ef8c-fc50-4c69-98cf-fc3a1f59d6c1-kube-api-access-d9vxm" pod "b683ef8c-fc50-4c69-98cf-fc3a1f59d6c1" (UID: "b683ef8c-fc50-4c69-98cf-fc3a1f59d6c1"). InnerVolumeSpecName "kube-api-access-d9vxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:14:25.228790 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:25.228751 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9vxm\" (UniqueName: \"kubernetes.io/projected/b683ef8c-fc50-4c69-98cf-fc3a1f59d6c1-kube-api-access-d9vxm\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:14:25.865345 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:25.865307 6221 generic.go:356] "Generic (PLEG): container finished" podID="b683ef8c-fc50-4c69-98cf-fc3a1f59d6c1" containerID="229e01cc8e36b137a5f9ff7d7d334862cbc6cfdd2db185e0d209f7cc71222c99" exitCode=0 Oct 30 02:14:25.865892 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:25.865349 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2eae39-d8f4c6966-szxzt" event={"ID":"b683ef8c-fc50-4c69-98cf-fc3a1f59d6c1","Type":"ContainerDied","Data":"229e01cc8e36b137a5f9ff7d7d334862cbc6cfdd2db185e0d209f7cc71222c99"} Oct 30 02:14:25.865892 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:25.865416 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-2eae39-d8f4c6966-szxzt" event={"ID":"b683ef8c-fc50-4c69-98cf-fc3a1f59d6c1","Type":"ContainerDied","Data":"83df210504d0299e34ccb40b9b1c82ce0b6fb7a6eca35564b43dfb8f431ef5f9"} Oct 30 02:14:25.865892 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:25.865439 6221 scope.go:122] "RemoveContainer" containerID="229e01cc8e36b137a5f9ff7d7d334862cbc6cfdd2db185e0d209f7cc71222c99" Oct 30 02:14:25.865892 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:25.865358 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-2eae39-d8f4c6966-szxzt" Oct 30 02:14:25.877524 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:25.877497 6221 scope.go:122] "RemoveContainer" containerID="229e01cc8e36b137a5f9ff7d7d334862cbc6cfdd2db185e0d209f7cc71222c99" Oct 30 02:14:25.877813 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:14:25.877792 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"229e01cc8e36b137a5f9ff7d7d334862cbc6cfdd2db185e0d209f7cc71222c99\": container with ID starting with 229e01cc8e36b137a5f9ff7d7d334862cbc6cfdd2db185e0d209f7cc71222c99 not found: ID does not exist" containerID="229e01cc8e36b137a5f9ff7d7d334862cbc6cfdd2db185e0d209f7cc71222c99" Oct 30 02:14:25.877890 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:25.877820 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"229e01cc8e36b137a5f9ff7d7d334862cbc6cfdd2db185e0d209f7cc71222c99"} err="failed to get container status \"229e01cc8e36b137a5f9ff7d7d334862cbc6cfdd2db185e0d209f7cc71222c99\": rpc error: code = NotFound desc = could not find container \"229e01cc8e36b137a5f9ff7d7d334862cbc6cfdd2db185e0d209f7cc71222c99\": container with ID starting with 229e01cc8e36b137a5f9ff7d7d334862cbc6cfdd2db185e0d209f7cc71222c99 not found: ID does not exist" Oct 30 02:14:25.919444 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:25.919414 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2eae39-d8f4c6966-szxzt"] Oct 30 02:14:25.931353 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:25.931329 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-2eae39-d8f4c6966-szxzt"] Oct 30 02:14:26.552189 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:26.552150 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b683ef8c-fc50-4c69-98cf-fc3a1f59d6c1" path="/var/lib/kubelet/pods/b683ef8c-fc50-4c69-98cf-fc3a1f59d6c1/volumes" Oct 30 02:14:33.107467 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:33.107425 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 02:14:51.495543 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:14:51.495494 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 02:14:57.062210 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:57.062171 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2eae39-58f6475475-r4tzv"] Oct 30 02:14:57.062712 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:57.062420 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2eae39-58f6475475-r4tzv" podUID="e13139d7-1bcd-4eed-9614-afa7194cc175" containerName="maas-api" containerID="cri-o://b869c5e6f02ec6dc07f6c5c513742b2526aa1c73c4106918ec78694d37db23cd" gracePeriod=30 Oct 30 02:14:57.549396 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:57.549061 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2eae39-58f6475475-r4tzv" Oct 30 02:14:57.636365 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:57.634629 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e13139d7-1bcd-4eed-9614-afa7194cc175-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e13139d7-1bcd-4eed-9614-afa7194cc175-openshift-service-ca\") pod \"e13139d7-1bcd-4eed-9614-afa7194cc175\" (UID: \"e13139d7-1bcd-4eed-9614-afa7194cc175\") " Oct 30 02:14:57.636365 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:57.634788 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e13139d7-1bcd-4eed-9614-afa7194cc175-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e13139d7-1bcd-4eed-9614-afa7194cc175-metrics-certs\") pod \"e13139d7-1bcd-4eed-9614-afa7194cc175\" (UID: \"e13139d7-1bcd-4eed-9614-afa7194cc175\") " Oct 30 02:14:57.636365 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:57.634829 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e13139d7-1bcd-4eed-9614-afa7194cc175-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e13139d7-1bcd-4eed-9614-afa7194cc175-maas-api-tls\") pod \"e13139d7-1bcd-4eed-9614-afa7194cc175\" (UID: \"e13139d7-1bcd-4eed-9614-afa7194cc175\") " Oct 30 02:14:57.636365 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:57.634971 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e13139d7-1bcd-4eed-9614-afa7194cc175-kube-api-access-p77gk\" (UniqueName: \"kubernetes.io/projected/e13139d7-1bcd-4eed-9614-afa7194cc175-kube-api-access-p77gk\") pod \"e13139d7-1bcd-4eed-9614-afa7194cc175\" (UID: \"e13139d7-1bcd-4eed-9614-afa7194cc175\") " Oct 30 02:14:57.638587 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:57.638561 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e13139d7-1bcd-4eed-9614-afa7194cc175-openshift-service-ca" pod "e13139d7-1bcd-4eed-9614-afa7194cc175" (UID: "e13139d7-1bcd-4eed-9614-afa7194cc175"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:14:57.654791 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:57.650128 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e13139d7-1bcd-4eed-9614-afa7194cc175-kube-api-access-p77gk" pod "e13139d7-1bcd-4eed-9614-afa7194cc175" (UID: "e13139d7-1bcd-4eed-9614-afa7194cc175"). InnerVolumeSpecName "kube-api-access-p77gk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:14:57.654791 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:57.650753 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e13139d7-1bcd-4eed-9614-afa7194cc175-metrics-certs" pod "e13139d7-1bcd-4eed-9614-afa7194cc175" (UID: "e13139d7-1bcd-4eed-9614-afa7194cc175"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:14:57.654791 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:57.652844 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e13139d7-1bcd-4eed-9614-afa7194cc175-maas-api-tls" pod "e13139d7-1bcd-4eed-9614-afa7194cc175" (UID: "e13139d7-1bcd-4eed-9614-afa7194cc175"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:14:57.737421 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:57.737317 6221 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e13139d7-1bcd-4eed-9614-afa7194cc175-openshift-service-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:14:57.737421 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:57.737360 6221 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e13139d7-1bcd-4eed-9614-afa7194cc175-metrics-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:14:57.737421 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:57.737370 6221 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e13139d7-1bcd-4eed-9614-afa7194cc175-maas-api-tls\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:14:57.737421 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:57.737378 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p77gk\" (UniqueName: \"kubernetes.io/projected/e13139d7-1bcd-4eed-9614-afa7194cc175-kube-api-access-p77gk\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:14:58.155590 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:58.155525 6221 generic.go:356] "Generic (PLEG): container finished" podID="e13139d7-1bcd-4eed-9614-afa7194cc175" containerID="b869c5e6f02ec6dc07f6c5c513742b2526aa1c73c4106918ec78694d37db23cd" exitCode=0 Oct 30 02:14:58.155590 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:58.155585 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2eae39-58f6475475-r4tzv" event={"ID":"e13139d7-1bcd-4eed-9614-afa7194cc175","Type":"ContainerDied","Data":"b869c5e6f02ec6dc07f6c5c513742b2526aa1c73c4106918ec78694d37db23cd"} Oct 30 02:14:58.156203 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:58.155603 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2eae39-58f6475475-r4tzv" Oct 30 02:14:58.156203 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:58.155644 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2eae39-58f6475475-r4tzv" event={"ID":"e13139d7-1bcd-4eed-9614-afa7194cc175","Type":"ContainerDied","Data":"aca52c833f511ace106eb432d1773b4af5af2a1a6cf374f58ab64fc0c3687c0e"} Oct 30 02:14:58.156203 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:58.155677 6221 scope.go:122] "RemoveContainer" containerID="b869c5e6f02ec6dc07f6c5c513742b2526aa1c73c4106918ec78694d37db23cd" Oct 30 02:14:58.177462 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:58.177425 6221 scope.go:122] "RemoveContainer" containerID="b869c5e6f02ec6dc07f6c5c513742b2526aa1c73c4106918ec78694d37db23cd" Oct 30 02:14:58.178039 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:14:58.177883 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b869c5e6f02ec6dc07f6c5c513742b2526aa1c73c4106918ec78694d37db23cd\": container with ID starting with b869c5e6f02ec6dc07f6c5c513742b2526aa1c73c4106918ec78694d37db23cd not found: ID does not exist" containerID="b869c5e6f02ec6dc07f6c5c513742b2526aa1c73c4106918ec78694d37db23cd" Oct 30 02:14:58.178039 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:58.177930 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b869c5e6f02ec6dc07f6c5c513742b2526aa1c73c4106918ec78694d37db23cd"} err="failed to get container status \"b869c5e6f02ec6dc07f6c5c513742b2526aa1c73c4106918ec78694d37db23cd\": rpc error: code = NotFound desc = could not find container \"b869c5e6f02ec6dc07f6c5c513742b2526aa1c73c4106918ec78694d37db23cd\": container with ID starting with b869c5e6f02ec6dc07f6c5c513742b2526aa1c73c4106918ec78694d37db23cd not found: ID does not exist" Oct 30 02:14:58.227086 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:58.227046 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2eae39-58f6475475-r4tzv"] Oct 30 02:14:58.237914 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:58.237879 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2eae39-58f6475475-r4tzv"] Oct 30 02:14:58.549759 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:14:58.549723 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e13139d7-1bcd-4eed-9614-afa7194cc175" path="/var/lib/kubelet/pods/e13139d7-1bcd-4eed-9614-afa7194cc175/volumes" Oct 30 02:15:00.227163 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.227134 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888775-tvzg8"] Oct 30 02:15:00.227973 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.227946 6221 cpu_manager.go:414] "containerMap: removing container" podUID="e13139d7-1bcd-4eed-9614-afa7194cc175" containerName="maas-api" Oct 30 02:15:00.227973 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.227974 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e13139d7-1bcd-4eed-9614-afa7194cc175" containerName="maas-api" Oct 30 02:15:00.228070 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.227991 6221 cpu_manager.go:414] "containerMap: removing container" podUID="b683ef8c-fc50-4c69-98cf-fc3a1f59d6c1" containerName="payload-processing" Oct 30 02:15:00.228070 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.227996 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b683ef8c-fc50-4c69-98cf-fc3a1f59d6c1" containerName="payload-processing" Oct 30 02:15:00.228070 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.228034 6221 cpu_manager.go:414] "containerMap: removing container" podUID="066ee435-d202-448a-ad09-17142e92ae5f" containerName="istio-proxy" Oct 30 02:15:00.228070 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.228039 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="066ee435-d202-448a-ad09-17142e92ae5f" containerName="istio-proxy" Oct 30 02:15:00.228255 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.228141 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="066ee435-d202-448a-ad09-17142e92ae5f" containerName="istio-proxy" Oct 30 02:15:00.228255 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.228153 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="e13139d7-1bcd-4eed-9614-afa7194cc175" containerName="maas-api" Oct 30 02:15:00.228255 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.228162 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="b683ef8c-fc50-4c69-98cf-fc3a1f59d6c1" containerName="payload-processing" Oct 30 02:15:00.231935 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.231912 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888775-tvzg8" Oct 30 02:15:00.245649 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.245222 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888775-x7l7p"] Oct 30 02:15:00.260786 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.260757 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888775-tvzg8"] Oct 30 02:15:00.261164 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.261142 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888775-x7l7p" Oct 30 02:15:00.267353 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.263700 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888775-jrp5d"] Oct 30 02:15:00.271619 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.270941 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888775-jrp5d" Oct 30 02:15:00.288129 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.284367 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888775-8zqzh"] Oct 30 02:15:00.288129 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.287243 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:15:00.290312 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.290288 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888775-8zqzh" Oct 30 02:15:00.294682 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.294624 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888775-x7l7p"] Oct 30 02:15:00.299284 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.299259 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888775-gdp5b"] Oct 30 02:15:00.311818 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.311793 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888775-jrp5d"] Oct 30 02:15:00.311962 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.311904 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888775-gdp5b" Oct 30 02:15:00.315312 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.315285 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888775-mwqtg"] Oct 30 02:15:00.324493 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.324466 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888775-8zqzh"] Oct 30 02:15:00.325232 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.324631 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888775-mwqtg" Oct 30 02:15:00.336417 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.335946 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-601d4b-29888775-d8mmg"] Oct 30 02:15:00.354911 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.354876 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-601d4b-29888775-d8mmg" Oct 30 02:15:00.365685 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.365647 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-aaf045-29888775-zkr6d"] Oct 30 02:15:00.376756 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.376727 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888775-gdp5b"] Oct 30 02:15:00.379219 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.377813 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-aaf045-29888775-zkr6d" Oct 30 02:15:00.382122 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.381481 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22e99d-29888775-kbvl4"] Oct 30 02:15:00.391049 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.391011 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-eda881-29888775-x7l7p\" (UID: \"cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888775-x7l7p" Oct 30 02:15:00.391185 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.391063 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/def2849e-c94c-4685-9992-e1121b744ac8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888775-8zqzh\" (UID: \"def2849e-c94c-4685-9992-e1121b744ac8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888775-8zqzh" Oct 30 02:15:00.391306 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.391285 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4ac5f29-bf87-42b2-82af-ee49cd16e879-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888775-tvzg8\" (UID: \"c4ac5f29-bf87-42b2-82af-ee49cd16e879\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888775-tvzg8" Oct 30 02:15:00.391388 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.391338 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66320d4a-0c8f-483d-98e4-b17658b6b27a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2e618f-29888775-jrp5d\" (UID: \"66320d4a-0c8f-483d-98e4-b17658b6b27a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888775-jrp5d" Oct 30 02:15:00.391455 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.391443 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvl5m\" (UniqueName: \"kubernetes.io/projected/cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9-kube-api-access-jvl5m\") pod \"maas-api-key-cleanup-e2e-worker-w2-eda881-29888775-x7l7p\" (UID: \"cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888775-x7l7p" Oct 30 02:15:00.391516 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.391478 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4st2\" (UniqueName: \"kubernetes.io/projected/c4ac5f29-bf87-42b2-82af-ee49cd16e879-kube-api-access-z4st2\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888775-tvzg8\" (UID: \"c4ac5f29-bf87-42b2-82af-ee49cd16e879\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888775-tvzg8" Oct 30 02:15:00.391655 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.391617 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h9fj\" (UniqueName: \"kubernetes.io/projected/def2849e-c94c-4685-9992-e1121b744ac8-kube-api-access-8h9fj\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888775-8zqzh\" (UID: \"def2849e-c94c-4685-9992-e1121b744ac8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888775-8zqzh" Oct 30 02:15:00.391788 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.391682 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdxjg\" (UniqueName: \"kubernetes.io/projected/66320d4a-0c8f-483d-98e4-b17658b6b27a-kube-api-access-mdxjg\") pod \"maas-api-key-cleanup-e2e-extmod-b-2e618f-29888775-jrp5d\" (UID: \"66320d4a-0c8f-483d-98e4-b17658b6b27a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888775-jrp5d" Oct 30 02:15:00.393587 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.393308 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888775-mwqtg"] Oct 30 02:15:00.393587 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.393410 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22e99d-29888775-kbvl4" Oct 30 02:15:00.395853 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.395834 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-aaf045-29888775-zkr6d"] Oct 30 02:15:00.399291 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.399266 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22e99d-29888775-kbvl4"] Oct 30 02:15:00.407159 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.407132 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-601d4b-29888775-d8mmg"] Oct 30 02:15:00.493230 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.493139 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68d8d09d-2725-417b-8d9d-c0ea49ce29ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888775-gdp5b\" (UID: \"68d8d09d-2725-417b-8d9d-c0ea49ce29ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888775-gdp5b" Oct 30 02:15:00.493230 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.493207 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff9b7\" (UniqueName: \"kubernetes.io/projected/ebe12490-67c0-4d1d-ad75-1ade885d5650-kube-api-access-ff9b7\") pod \"maas-api-key-cleanup-e2e-inf-b-601d4b-29888775-d8mmg\" (UID: \"ebe12490-67c0-4d1d-ad75-1ade885d5650\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-601d4b-29888775-d8mmg" Oct 30 02:15:00.493485 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.493287 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-795f8\" (UniqueName: \"kubernetes.io/projected/68d8d09d-2725-417b-8d9d-c0ea49ce29ac-kube-api-access-795f8\") pod \"maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888775-gdp5b\" (UID: \"68d8d09d-2725-417b-8d9d-c0ea49ce29ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888775-gdp5b" Oct 30 02:15:00.493485 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.493348 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-eda881-29888775-x7l7p\" (UID: \"cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888775-x7l7p" Oct 30 02:15:00.493485 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.493391 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/def2849e-c94c-4685-9992-e1121b744ac8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888775-8zqzh\" (UID: \"def2849e-c94c-4685-9992-e1121b744ac8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888775-8zqzh" Oct 30 02:15:00.493485 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.493429 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe12490-67c0-4d1d-ad75-1ade885d5650-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-601d4b-29888775-d8mmg\" (UID: \"ebe12490-67c0-4d1d-ad75-1ade885d5650\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-601d4b-29888775-d8mmg" Oct 30 02:15:00.493485 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.493482 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vbqt\" (UniqueName: \"kubernetes.io/projected/a506dae9-1ced-409d-99db-413b66dd9d59-kube-api-access-4vbqt\") pod \"maas-api-key-cleanup-e2e-inf-a-aaf045-29888775-zkr6d\" (UID: \"a506dae9-1ced-409d-99db-413b66dd9d59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-aaf045-29888775-zkr6d" Oct 30 02:15:00.493790 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.493575 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/144c527d-0189-4c40-a64a-a207dfbdd710-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-76a19e-29888775-mwqtg\" (UID: \"144c527d-0189-4c40-a64a-a207dfbdd710\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888775-mwqtg" Oct 30 02:15:00.493790 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.493605 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a506dae9-1ced-409d-99db-413b66dd9d59-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-aaf045-29888775-zkr6d\" (UID: \"a506dae9-1ced-409d-99db-413b66dd9d59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-aaf045-29888775-zkr6d" Oct 30 02:15:00.493790 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.493694 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4ac5f29-bf87-42b2-82af-ee49cd16e879-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888775-tvzg8\" (UID: \"c4ac5f29-bf87-42b2-82af-ee49cd16e879\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888775-tvzg8" Oct 30 02:15:00.493790 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.493716 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnsdb\" (UniqueName: \"kubernetes.io/projected/d90c1ed4-5a28-4ce1-ba6e-c521698c3667-kube-api-access-nnsdb\") pod \"maas-api-key-cleanup-e2e-inf-b-22e99d-29888775-kbvl4\" (UID: \"d90c1ed4-5a28-4ce1-ba6e-c521698c3667\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22e99d-29888775-kbvl4" Oct 30 02:15:00.493790 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.493744 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66320d4a-0c8f-483d-98e4-b17658b6b27a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2e618f-29888775-jrp5d\" (UID: \"66320d4a-0c8f-483d-98e4-b17658b6b27a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888775-jrp5d" Oct 30 02:15:00.494067 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.493873 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgqvk\" (UniqueName: \"kubernetes.io/projected/144c527d-0189-4c40-a64a-a207dfbdd710-kube-api-access-xgqvk\") pod \"maas-api-key-cleanup-e2e-extmod-a-76a19e-29888775-mwqtg\" (UID: \"144c527d-0189-4c40-a64a-a207dfbdd710\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888775-mwqtg" Oct 30 02:15:00.494067 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.493910 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvl5m\" (UniqueName: \"kubernetes.io/projected/cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9-kube-api-access-jvl5m\") pod \"maas-api-key-cleanup-e2e-worker-w2-eda881-29888775-x7l7p\" (UID: \"cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888775-x7l7p" Oct 30 02:15:00.494067 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.493962 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4st2\" (UniqueName: \"kubernetes.io/projected/c4ac5f29-bf87-42b2-82af-ee49cd16e879-kube-api-access-z4st2\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888775-tvzg8\" (UID: \"c4ac5f29-bf87-42b2-82af-ee49cd16e879\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888775-tvzg8" Oct 30 02:15:00.494067 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.494032 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d90c1ed4-5a28-4ce1-ba6e-c521698c3667-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-22e99d-29888775-kbvl4\" (UID: \"d90c1ed4-5a28-4ce1-ba6e-c521698c3667\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22e99d-29888775-kbvl4" Oct 30 02:15:00.494352 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.494259 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h9fj\" (UniqueName: \"kubernetes.io/projected/def2849e-c94c-4685-9992-e1121b744ac8-kube-api-access-8h9fj\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888775-8zqzh\" (UID: \"def2849e-c94c-4685-9992-e1121b744ac8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888775-8zqzh" Oct 30 02:15:00.494352 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.494318 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdxjg\" (UniqueName: \"kubernetes.io/projected/66320d4a-0c8f-483d-98e4-b17658b6b27a-kube-api-access-mdxjg\") pod \"maas-api-key-cleanup-e2e-extmod-b-2e618f-29888775-jrp5d\" (UID: \"66320d4a-0c8f-483d-98e4-b17658b6b27a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888775-jrp5d" Oct 30 02:15:00.494469 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.494442 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/def2849e-c94c-4685-9992-e1121b744ac8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888775-8zqzh\" (UID: \"def2849e-c94c-4685-9992-e1121b744ac8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888775-8zqzh" Oct 30 02:15:00.494603 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.494580 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-eda881-29888775-x7l7p\" (UID: \"cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888775-x7l7p" Oct 30 02:15:00.494826 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.494802 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66320d4a-0c8f-483d-98e4-b17658b6b27a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2e618f-29888775-jrp5d\" (UID: \"66320d4a-0c8f-483d-98e4-b17658b6b27a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888775-jrp5d" Oct 30 02:15:00.494923 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.494905 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4ac5f29-bf87-42b2-82af-ee49cd16e879-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888775-tvzg8\" (UID: \"c4ac5f29-bf87-42b2-82af-ee49cd16e879\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888775-tvzg8" Oct 30 02:15:00.513872 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.513843 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h9fj\" (UniqueName: \"kubernetes.io/projected/def2849e-c94c-4685-9992-e1121b744ac8-kube-api-access-8h9fj\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888775-8zqzh\" (UID: \"def2849e-c94c-4685-9992-e1121b744ac8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888775-8zqzh" Oct 30 02:15:00.514029 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.513877 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdxjg\" (UniqueName: \"kubernetes.io/projected/66320d4a-0c8f-483d-98e4-b17658b6b27a-kube-api-access-mdxjg\") pod \"maas-api-key-cleanup-e2e-extmod-b-2e618f-29888775-jrp5d\" (UID: \"66320d4a-0c8f-483d-98e4-b17658b6b27a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888775-jrp5d" Oct 30 02:15:00.514029 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.514020 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvl5m\" (UniqueName: \"kubernetes.io/projected/cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9-kube-api-access-jvl5m\") pod \"maas-api-key-cleanup-e2e-worker-w2-eda881-29888775-x7l7p\" (UID: \"cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888775-x7l7p" Oct 30 02:15:00.514120 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.514034 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4st2\" (UniqueName: \"kubernetes.io/projected/c4ac5f29-bf87-42b2-82af-ee49cd16e879-kube-api-access-z4st2\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888775-tvzg8\" (UID: \"c4ac5f29-bf87-42b2-82af-ee49cd16e879\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888775-tvzg8" Oct 30 02:15:00.553593 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.553561 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888775-tvzg8" Oct 30 02:15:00.590946 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.590916 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888775-x7l7p" Oct 30 02:15:00.595288 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.595260 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68d8d09d-2725-417b-8d9d-c0ea49ce29ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888775-gdp5b\" (UID: \"68d8d09d-2725-417b-8d9d-c0ea49ce29ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888775-gdp5b" Oct 30 02:15:00.595400 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.595352 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff9b7\" (UniqueName: \"kubernetes.io/projected/ebe12490-67c0-4d1d-ad75-1ade885d5650-kube-api-access-ff9b7\") pod \"maas-api-key-cleanup-e2e-inf-b-601d4b-29888775-d8mmg\" (UID: \"ebe12490-67c0-4d1d-ad75-1ade885d5650\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-601d4b-29888775-d8mmg" Oct 30 02:15:00.595471 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.595396 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-795f8\" (UniqueName: \"kubernetes.io/projected/68d8d09d-2725-417b-8d9d-c0ea49ce29ac-kube-api-access-795f8\") pod \"maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888775-gdp5b\" (UID: \"68d8d09d-2725-417b-8d9d-c0ea49ce29ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888775-gdp5b" Oct 30 02:15:00.595629 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.595474 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe12490-67c0-4d1d-ad75-1ade885d5650-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-601d4b-29888775-d8mmg\" (UID: \"ebe12490-67c0-4d1d-ad75-1ade885d5650\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-601d4b-29888775-d8mmg" Oct 30 02:15:00.595629 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.595596 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vbqt\" (UniqueName: \"kubernetes.io/projected/a506dae9-1ced-409d-99db-413b66dd9d59-kube-api-access-4vbqt\") pod \"maas-api-key-cleanup-e2e-inf-a-aaf045-29888775-zkr6d\" (UID: \"a506dae9-1ced-409d-99db-413b66dd9d59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-aaf045-29888775-zkr6d" Oct 30 02:15:00.595775 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.595635 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/144c527d-0189-4c40-a64a-a207dfbdd710-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-76a19e-29888775-mwqtg\" (UID: \"144c527d-0189-4c40-a64a-a207dfbdd710\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888775-mwqtg" Oct 30 02:15:00.595775 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.595664 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a506dae9-1ced-409d-99db-413b66dd9d59-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-aaf045-29888775-zkr6d\" (UID: \"a506dae9-1ced-409d-99db-413b66dd9d59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-aaf045-29888775-zkr6d" Oct 30 02:15:00.595775 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.595748 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnsdb\" (UniqueName: \"kubernetes.io/projected/d90c1ed4-5a28-4ce1-ba6e-c521698c3667-kube-api-access-nnsdb\") pod \"maas-api-key-cleanup-e2e-inf-b-22e99d-29888775-kbvl4\" (UID: \"d90c1ed4-5a28-4ce1-ba6e-c521698c3667\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22e99d-29888775-kbvl4" Oct 30 02:15:00.595933 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.595834 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgqvk\" (UniqueName: \"kubernetes.io/projected/144c527d-0189-4c40-a64a-a207dfbdd710-kube-api-access-xgqvk\") pod \"maas-api-key-cleanup-e2e-extmod-a-76a19e-29888775-mwqtg\" (UID: \"144c527d-0189-4c40-a64a-a207dfbdd710\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888775-mwqtg" Oct 30 02:15:00.595933 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.595904 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d90c1ed4-5a28-4ce1-ba6e-c521698c3667-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-22e99d-29888775-kbvl4\" (UID: \"d90c1ed4-5a28-4ce1-ba6e-c521698c3667\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22e99d-29888775-kbvl4" Oct 30 02:15:00.596116 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.596088 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68d8d09d-2725-417b-8d9d-c0ea49ce29ac-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888775-gdp5b\" (UID: \"68d8d09d-2725-417b-8d9d-c0ea49ce29ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888775-gdp5b" Oct 30 02:15:00.596615 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.596589 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe12490-67c0-4d1d-ad75-1ade885d5650-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-601d4b-29888775-d8mmg\" (UID: \"ebe12490-67c0-4d1d-ad75-1ade885d5650\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-601d4b-29888775-d8mmg" Oct 30 02:15:00.596730 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.596654 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d90c1ed4-5a28-4ce1-ba6e-c521698c3667-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-22e99d-29888775-kbvl4\" (UID: \"d90c1ed4-5a28-4ce1-ba6e-c521698c3667\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22e99d-29888775-kbvl4" Oct 30 02:15:00.596730 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.596715 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a506dae9-1ced-409d-99db-413b66dd9d59-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-aaf045-29888775-zkr6d\" (UID: \"a506dae9-1ced-409d-99db-413b66dd9d59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-aaf045-29888775-zkr6d" Oct 30 02:15:00.596851 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.596837 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/144c527d-0189-4c40-a64a-a207dfbdd710-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-76a19e-29888775-mwqtg\" (UID: \"144c527d-0189-4c40-a64a-a207dfbdd710\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888775-mwqtg" Oct 30 02:15:00.605577 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.604085 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888775-jrp5d" Oct 30 02:15:00.620722 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.620697 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff9b7\" (UniqueName: \"kubernetes.io/projected/ebe12490-67c0-4d1d-ad75-1ade885d5650-kube-api-access-ff9b7\") pod \"maas-api-key-cleanup-e2e-inf-b-601d4b-29888775-d8mmg\" (UID: \"ebe12490-67c0-4d1d-ad75-1ade885d5650\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-601d4b-29888775-d8mmg" Oct 30 02:15:00.625224 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.625051 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vbqt\" (UniqueName: \"kubernetes.io/projected/a506dae9-1ced-409d-99db-413b66dd9d59-kube-api-access-4vbqt\") pod \"maas-api-key-cleanup-e2e-inf-a-aaf045-29888775-zkr6d\" (UID: \"a506dae9-1ced-409d-99db-413b66dd9d59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-aaf045-29888775-zkr6d" Oct 30 02:15:00.625349 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.625295 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888775-8zqzh" Oct 30 02:15:00.626250 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.626220 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgqvk\" (UniqueName: \"kubernetes.io/projected/144c527d-0189-4c40-a64a-a207dfbdd710-kube-api-access-xgqvk\") pod \"maas-api-key-cleanup-e2e-extmod-a-76a19e-29888775-mwqtg\" (UID: \"144c527d-0189-4c40-a64a-a207dfbdd710\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888775-mwqtg" Oct 30 02:15:00.626365 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.626323 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-795f8\" (UniqueName: \"kubernetes.io/projected/68d8d09d-2725-417b-8d9d-c0ea49ce29ac-kube-api-access-795f8\") pod \"maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888775-gdp5b\" (UID: \"68d8d09d-2725-417b-8d9d-c0ea49ce29ac\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888775-gdp5b" Oct 30 02:15:00.629837 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.629768 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnsdb\" (UniqueName: \"kubernetes.io/projected/d90c1ed4-5a28-4ce1-ba6e-c521698c3667-kube-api-access-nnsdb\") pod \"maas-api-key-cleanup-e2e-inf-b-22e99d-29888775-kbvl4\" (UID: \"d90c1ed4-5a28-4ce1-ba6e-c521698c3667\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22e99d-29888775-kbvl4" Oct 30 02:15:00.667787 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.667754 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888775-gdp5b" Oct 30 02:15:00.677957 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.677925 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888775-mwqtg" Oct 30 02:15:00.703970 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.702269 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-601d4b-29888775-d8mmg" Oct 30 02:15:00.709247 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.707854 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-aaf045-29888775-zkr6d" Oct 30 02:15:00.713233 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.713206 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22e99d-29888775-kbvl4" Oct 30 02:15:00.869114 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:00.868322 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888775-tvzg8"] Oct 30 02:15:00.874008 ip-10-0-135-65 kubenswrapper[6221]: W1030 02:15:00.873981 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4ac5f29_bf87_42b2_82af_ee49cd16e879.slice/crio-9930c68c5c436977368c934870754a40f2f58cd88502653487fbf5625d517273 WatchSource:0}: Error finding container 9930c68c5c436977368c934870754a40f2f58cd88502653487fbf5625d517273: Status 404 returned error can't find the container with id 9930c68c5c436977368c934870754a40f2f58cd88502653487fbf5625d517273 Oct 30 02:15:01.081162 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:01.081134 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888775-mwqtg"] Oct 30 02:15:01.083277 ip-10-0-135-65 kubenswrapper[6221]: W1030 02:15:01.083247 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod144c527d_0189_4c40_a64a_a207dfbdd710.slice/crio-118c846dc6663fecafd3511c728520893787b59c2954ad67df3ddbc0de8915f0 WatchSource:0}: Error finding container 118c846dc6663fecafd3511c728520893787b59c2954ad67df3ddbc0de8915f0: Status 404 returned error can't find the container with id 118c846dc6663fecafd3511c728520893787b59c2954ad67df3ddbc0de8915f0 Oct 30 02:15:01.104498 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:01.104469 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-601d4b-29888775-d8mmg"] Oct 30 02:15:01.105712 ip-10-0-135-65 kubenswrapper[6221]: W1030 02:15:01.105683 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebe12490_67c0_4d1d_ad75_1ade885d5650.slice/crio-3536aa7411e3ed92a3c7c9daac5627b596f5e03e954b6b1609837575c3b62b23 WatchSource:0}: Error finding container 3536aa7411e3ed92a3c7c9daac5627b596f5e03e954b6b1609837575c3b62b23: Status 404 returned error can't find the container with id 3536aa7411e3ed92a3c7c9daac5627b596f5e03e954b6b1609837575c3b62b23 Oct 30 02:15:01.115439 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:01.115414 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-aaf045-29888775-zkr6d"] Oct 30 02:15:01.119842 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:01.119814 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22e99d-29888775-kbvl4"] Oct 30 02:15:01.124373 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:01.124349 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888775-x7l7p"] Oct 30 02:15:01.126839 ip-10-0-135-65 kubenswrapper[6221]: W1030 02:15:01.126816 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc0a37d6_fc06_482d_abda_0f1d0ffcc2f9.slice/crio-19d8fed769322b7c48c82dd8376f5a64ee5ec60342f25f9de9f27e49acfe750e WatchSource:0}: Error finding container 19d8fed769322b7c48c82dd8376f5a64ee5ec60342f25f9de9f27e49acfe750e: Status 404 returned error can't find the container with id 19d8fed769322b7c48c82dd8376f5a64ee5ec60342f25f9de9f27e49acfe750e Oct 30 02:15:01.146351 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:01.146329 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888775-jrp5d"] Oct 30 02:15:01.160008 ip-10-0-135-65 kubenswrapper[6221]: W1030 02:15:01.159984 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66320d4a_0c8f_483d_98e4_b17658b6b27a.slice/crio-93da2a896cb5e6f491bfd2e64cf3e624d8cec3eb32fe5785a4da09a3777a94ab WatchSource:0}: Error finding container 93da2a896cb5e6f491bfd2e64cf3e624d8cec3eb32fe5785a4da09a3777a94ab: Status 404 returned error can't find the container with id 93da2a896cb5e6f491bfd2e64cf3e624d8cec3eb32fe5785a4da09a3777a94ab Oct 30 02:15:01.181450 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:01.181415 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-601d4b-29888775-d8mmg" event={"ID":"ebe12490-67c0-4d1d-ad75-1ade885d5650","Type":"ContainerStarted","Data":"3536aa7411e3ed92a3c7c9daac5627b596f5e03e954b6b1609837575c3b62b23"} Oct 30 02:15:01.183110 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:01.183039 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888775-mwqtg" event={"ID":"144c527d-0189-4c40-a64a-a207dfbdd710","Type":"ContainerStarted","Data":"118c846dc6663fecafd3511c728520893787b59c2954ad67df3ddbc0de8915f0"} Oct 30 02:15:01.184862 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:01.184648 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888775-tvzg8" event={"ID":"c4ac5f29-bf87-42b2-82af-ee49cd16e879","Type":"ContainerStarted","Data":"9930c68c5c436977368c934870754a40f2f58cd88502653487fbf5625d517273"} Oct 30 02:15:01.186074 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:01.186049 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-aaf045-29888775-zkr6d" event={"ID":"a506dae9-1ced-409d-99db-413b66dd9d59","Type":"ContainerStarted","Data":"d13a85c746e9976b87e7f70055baf4f5cbfcbc58b79b1a8fd1ff4f054820ab83"} Oct 30 02:15:01.187443 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:01.187422 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888775-jrp5d" event={"ID":"66320d4a-0c8f-483d-98e4-b17658b6b27a","Type":"ContainerStarted","Data":"93da2a896cb5e6f491bfd2e64cf3e624d8cec3eb32fe5785a4da09a3777a94ab"} Oct 30 02:15:01.189229 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:01.189094 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22e99d-29888775-kbvl4" event={"ID":"d90c1ed4-5a28-4ce1-ba6e-c521698c3667","Type":"ContainerStarted","Data":"ebec57363ced8d43445e91b2e7671b7b5c87a983613afb0dd7099687ce4a16f6"} Oct 30 02:15:01.190767 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:01.190745 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888775-x7l7p" event={"ID":"cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9","Type":"ContainerStarted","Data":"19d8fed769322b7c48c82dd8376f5a64ee5ec60342f25f9de9f27e49acfe750e"} Oct 30 02:15:01.194868 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:01.194842 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888775-8zqzh"] Oct 30 02:15:01.196901 ip-10-0-135-65 kubenswrapper[6221]: W1030 02:15:01.196876 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddef2849e_c94c_4685_9992_e1121b744ac8.slice/crio-18ccb5511a535f22e4bf02f93123c92687cb202fc1bc4930a4d3c56d24291c7f WatchSource:0}: Error finding container 18ccb5511a535f22e4bf02f93123c92687cb202fc1bc4930a4d3c56d24291c7f: Status 404 returned error can't find the container with id 18ccb5511a535f22e4bf02f93123c92687cb202fc1bc4930a4d3c56d24291c7f Oct 30 02:15:01.199237 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:01.199216 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4"] Oct 30 02:15:01.199405 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:01.199385 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4" podUID="de9bc101-0002-4af2-a1fa-ae9c6be2ff73" containerName="istio-proxy" containerID="cri-o://7750b5c6e89c2dce8a4b1576b879fcdc33affd79f32bb3879266be3ac216d8d3" gracePeriod=30 Oct 30 02:15:01.241038 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:01.241005 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888775-gdp5b"] Oct 30 02:15:01.328075 ip-10-0-135-65 kubenswrapper[6221]: W1030 02:15:01.328043 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68d8d09d_2725_417b_8d9d_c0ea49ce29ac.slice/crio-246e07c750089299f50e207133ead594a4d9e2d424459248eef3f287d3641f1c WatchSource:0}: Error finding container 246e07c750089299f50e207133ead594a4d9e2d424459248eef3f287d3641f1c: Status 404 returned error can't find the container with id 246e07c750089299f50e207133ead594a4d9e2d424459248eef3f287d3641f1c Oct 30 02:15:02.201024 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:02.200914 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22e99d-29888775-kbvl4" event={"ID":"d90c1ed4-5a28-4ce1-ba6e-c521698c3667","Type":"ContainerStarted","Data":"c9b64aca6d2da8cdbd3d6c08797bcee0134167d9ded79a9f35ef53c16bde9b54"} Oct 30 02:15:02.202673 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:02.202636 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888775-8zqzh" event={"ID":"def2849e-c94c-4685-9992-e1121b744ac8","Type":"ContainerStarted","Data":"6904aad63a54c111480fcca65ecef25893cb780f44a79009e79b9750ed20c5cc"} Oct 30 02:15:02.202814 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:02.202696 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888775-8zqzh" event={"ID":"def2849e-c94c-4685-9992-e1121b744ac8","Type":"ContainerStarted","Data":"18ccb5511a535f22e4bf02f93123c92687cb202fc1bc4930a4d3c56d24291c7f"} Oct 30 02:15:02.204261 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:02.204235 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888775-x7l7p" event={"ID":"cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9","Type":"ContainerStarted","Data":"ee78fd27ea6af171734f00551887c2d5e6ec82929a149dcdd904fe1637ec0d1f"} Oct 30 02:15:02.205811 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:02.205788 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-601d4b-29888775-d8mmg" event={"ID":"ebe12490-67c0-4d1d-ad75-1ade885d5650","Type":"ContainerStarted","Data":"21081a101abbc5aef195963a3397569ed035eeb582668f03fa60a031915ec78a"} Oct 30 02:15:02.209325 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:02.209296 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888775-gdp5b" event={"ID":"68d8d09d-2725-417b-8d9d-c0ea49ce29ac","Type":"ContainerStarted","Data":"f47cb1f76bbe425a0497bcedc239c6c382c9a4cb0ca37c393ddc806afa5fb6be"} Oct 30 02:15:02.209515 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:02.209484 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888775-gdp5b" event={"ID":"68d8d09d-2725-417b-8d9d-c0ea49ce29ac","Type":"ContainerStarted","Data":"246e07c750089299f50e207133ead594a4d9e2d424459248eef3f287d3641f1c"} Oct 30 02:15:02.212203 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:02.212173 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888775-mwqtg" event={"ID":"144c527d-0189-4c40-a64a-a207dfbdd710","Type":"ContainerStarted","Data":"04ba3e18e0ea3f6ca5538a8c249fc98696a2e3d6c33d04ff8ff358a42c73e9c3"} Oct 30 02:15:02.215147 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:02.215118 6221 generic.go:356] "Generic (PLEG): container finished" podID="c4ac5f29-bf87-42b2-82af-ee49cd16e879" containerID="0dabc5db7f0ba80b2d1a5da7f7fa9d80ebf0400448a7acbb2d1f88a7a61216bd" exitCode=60 Oct 30 02:15:02.215301 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:02.215280 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888775-tvzg8" event={"ID":"c4ac5f29-bf87-42b2-82af-ee49cd16e879","Type":"ContainerDied","Data":"0dabc5db7f0ba80b2d1a5da7f7fa9d80ebf0400448a7acbb2d1f88a7a61216bd"} Oct 30 02:15:02.215489 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:02.215471 6221 scope.go:122] "RemoveContainer" containerID="0dabc5db7f0ba80b2d1a5da7f7fa9d80ebf0400448a7acbb2d1f88a7a61216bd" Oct 30 02:15:02.218409 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:02.218382 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-aaf045-29888775-zkr6d" event={"ID":"a506dae9-1ced-409d-99db-413b66dd9d59","Type":"ContainerStarted","Data":"acd3b800d049fde420e28d06f0b127aa6d171b77c4cc484c0a8ed9271106bc3b"} Oct 30 02:15:02.220394 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:02.220371 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888775-jrp5d" event={"ID":"66320d4a-0c8f-483d-98e4-b17658b6b27a","Type":"ContainerStarted","Data":"3ebd3bede478a95c7d5840896c1f2bbf56e06d146f04b3699816e7d64d37f59f"} Oct 30 02:15:02.242631 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:02.242571 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22e99d-29888775-kbvl4" podStartSLOduration=2.242519535 podStartE2EDuration="2.242519535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:02.239627517 +0000 UTC m=+3425.817914879" watchObservedRunningTime="2026-10-30 02:15:02.242519535 +0000 UTC m=+3425.820806872" Oct 30 02:15:02.282575 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:02.282493 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-601d4b-29888775-d8mmg" podStartSLOduration=2.282476242 podStartE2EDuration="2.282476242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:02.280402306 +0000 UTC m=+3425.858689642" watchObservedRunningTime="2026-10-30 02:15:02.282476242 +0000 UTC m=+3425.860763581" Oct 30 02:15:02.323866 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:02.323811 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888775-8zqzh" podStartSLOduration=2.323796989 podStartE2EDuration="2.323796989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:02.318132493 +0000 UTC m=+3425.896419831" watchObservedRunningTime="2026-10-30 02:15:02.323796989 +0000 UTC m=+3425.902084316" Oct 30 02:15:02.359734 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:02.359679 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888775-x7l7p" podStartSLOduration=2.359665736 podStartE2EDuration="2.359665736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:02.356097075 +0000 UTC m=+3425.934384415" watchObservedRunningTime="2026-10-30 02:15:02.359665736 +0000 UTC m=+3425.937953071" Oct 30 02:15:02.395838 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:02.395775 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888775-jrp5d" podStartSLOduration=2.395757917 podStartE2EDuration="2.395757917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:02.393607102 +0000 UTC m=+3425.971894440" watchObservedRunningTime="2026-10-30 02:15:02.395757917 +0000 UTC m=+3425.974045256" Oct 30 02:15:02.436073 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:02.436020 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888775-gdp5b" podStartSLOduration=2.436005332 podStartE2EDuration="2.436005332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:02.432609819 +0000 UTC m=+3426.010897156" watchObservedRunningTime="2026-10-30 02:15:02.436005332 +0000 UTC m=+3426.014292667" Oct 30 02:15:02.512277 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:02.512004 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888775-mwqtg" podStartSLOduration=2.511985219 podStartE2EDuration="2.511985219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:02.510085633 +0000 UTC m=+3426.088372966" watchObservedRunningTime="2026-10-30 02:15:02.511985219 +0000 UTC m=+3426.090272556" Oct 30 02:15:02.552247 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:02.552179 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-aaf045-29888775-zkr6d" podStartSLOduration=2.552163801 podStartE2EDuration="2.552163801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:02.551700246 +0000 UTC m=+3426.129987583" watchObservedRunningTime="2026-10-30 02:15:02.552163801 +0000 UTC m=+3426.130451167" Oct 30 02:15:03.232046 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:03.232013 6221 generic.go:356] "Generic (PLEG): container finished" podID="def2849e-c94c-4685-9992-e1121b744ac8" containerID="6904aad63a54c111480fcca65ecef25893cb780f44a79009e79b9750ed20c5cc" exitCode=60 Oct 30 02:15:03.232239 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:03.232100 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888775-8zqzh" event={"ID":"def2849e-c94c-4685-9992-e1121b744ac8","Type":"ContainerDied","Data":"6904aad63a54c111480fcca65ecef25893cb780f44a79009e79b9750ed20c5cc"} Oct 30 02:15:03.232459 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:03.232440 6221 scope.go:122] "RemoveContainer" containerID="6904aad63a54c111480fcca65ecef25893cb780f44a79009e79b9750ed20c5cc" Oct 30 02:15:03.233676 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:03.233652 6221 generic.go:356] "Generic (PLEG): container finished" podID="cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9" containerID="ee78fd27ea6af171734f00551887c2d5e6ec82929a149dcdd904fe1637ec0d1f" exitCode=60 Oct 30 02:15:03.233791 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:03.233736 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888775-x7l7p" event={"ID":"cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9","Type":"ContainerDied","Data":"ee78fd27ea6af171734f00551887c2d5e6ec82929a149dcdd904fe1637ec0d1f"} Oct 30 02:15:03.234004 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:03.233982 6221 scope.go:122] "RemoveContainer" containerID="ee78fd27ea6af171734f00551887c2d5e6ec82929a149dcdd904fe1637ec0d1f" Oct 30 02:15:03.236018 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:03.235998 6221 generic.go:356] "Generic (PLEG): container finished" podID="ebe12490-67c0-4d1d-ad75-1ade885d5650" containerID="21081a101abbc5aef195963a3397569ed035eeb582668f03fa60a031915ec78a" exitCode=60 Oct 30 02:15:03.236129 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:03.236109 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-601d4b-29888775-d8mmg" event={"ID":"ebe12490-67c0-4d1d-ad75-1ade885d5650","Type":"ContainerDied","Data":"21081a101abbc5aef195963a3397569ed035eeb582668f03fa60a031915ec78a"} Oct 30 02:15:03.236321 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:03.236307 6221 scope.go:122] "RemoveContainer" containerID="21081a101abbc5aef195963a3397569ed035eeb582668f03fa60a031915ec78a" Oct 30 02:15:03.237794 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:03.237750 6221 generic.go:356] "Generic (PLEG): container finished" podID="68d8d09d-2725-417b-8d9d-c0ea49ce29ac" containerID="f47cb1f76bbe425a0497bcedc239c6c382c9a4cb0ca37c393ddc806afa5fb6be" exitCode=60 Oct 30 02:15:03.237863 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:03.237836 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888775-gdp5b" event={"ID":"68d8d09d-2725-417b-8d9d-c0ea49ce29ac","Type":"ContainerDied","Data":"f47cb1f76bbe425a0497bcedc239c6c382c9a4cb0ca37c393ddc806afa5fb6be"} Oct 30 02:15:03.238038 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:03.238023 6221 scope.go:122] "RemoveContainer" containerID="f47cb1f76bbe425a0497bcedc239c6c382c9a4cb0ca37c393ddc806afa5fb6be" Oct 30 02:15:03.239519 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:03.239498 6221 generic.go:356] "Generic (PLEG): container finished" podID="144c527d-0189-4c40-a64a-a207dfbdd710" containerID="04ba3e18e0ea3f6ca5538a8c249fc98696a2e3d6c33d04ff8ff358a42c73e9c3" exitCode=60 Oct 30 02:15:03.239608 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:03.239554 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888775-mwqtg" event={"ID":"144c527d-0189-4c40-a64a-a207dfbdd710","Type":"ContainerDied","Data":"04ba3e18e0ea3f6ca5538a8c249fc98696a2e3d6c33d04ff8ff358a42c73e9c3"} Oct 30 02:15:03.239819 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:03.239800 6221 scope.go:122] "RemoveContainer" containerID="04ba3e18e0ea3f6ca5538a8c249fc98696a2e3d6c33d04ff8ff358a42c73e9c3" Oct 30 02:15:03.241691 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:03.241665 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888775-tvzg8" event={"ID":"c4ac5f29-bf87-42b2-82af-ee49cd16e879","Type":"ContainerStarted","Data":"5de1ee88759862400a5bd827020612f9896294dcd9118f9aa905ecc23b6dda23"} Oct 30 02:15:03.243288 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:03.243237 6221 generic.go:356] "Generic (PLEG): container finished" podID="a506dae9-1ced-409d-99db-413b66dd9d59" containerID="acd3b800d049fde420e28d06f0b127aa6d171b77c4cc484c0a8ed9271106bc3b" exitCode=60 Oct 30 02:15:03.243288 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:03.243267 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-aaf045-29888775-zkr6d" event={"ID":"a506dae9-1ced-409d-99db-413b66dd9d59","Type":"ContainerDied","Data":"acd3b800d049fde420e28d06f0b127aa6d171b77c4cc484c0a8ed9271106bc3b"} Oct 30 02:15:03.243806 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:03.243515 6221 scope.go:122] "RemoveContainer" containerID="acd3b800d049fde420e28d06f0b127aa6d171b77c4cc484c0a8ed9271106bc3b" Oct 30 02:15:03.246335 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:03.246270 6221 generic.go:356] "Generic (PLEG): container finished" podID="66320d4a-0c8f-483d-98e4-b17658b6b27a" containerID="3ebd3bede478a95c7d5840896c1f2bbf56e06d146f04b3699816e7d64d37f59f" exitCode=60 Oct 30 02:15:03.246625 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:03.246394 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888775-jrp5d" event={"ID":"66320d4a-0c8f-483d-98e4-b17658b6b27a","Type":"ContainerDied","Data":"3ebd3bede478a95c7d5840896c1f2bbf56e06d146f04b3699816e7d64d37f59f"} Oct 30 02:15:03.246901 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:03.246868 6221 scope.go:122] "RemoveContainer" containerID="3ebd3bede478a95c7d5840896c1f2bbf56e06d146f04b3699816e7d64d37f59f" Oct 30 02:15:03.248604 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:03.248579 6221 generic.go:356] "Generic (PLEG): container finished" podID="d90c1ed4-5a28-4ce1-ba6e-c521698c3667" containerID="c9b64aca6d2da8cdbd3d6c08797bcee0134167d9ded79a9f35ef53c16bde9b54" exitCode=60 Oct 30 02:15:03.248685 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:03.248642 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22e99d-29888775-kbvl4" event={"ID":"d90c1ed4-5a28-4ce1-ba6e-c521698c3667","Type":"ContainerDied","Data":"c9b64aca6d2da8cdbd3d6c08797bcee0134167d9ded79a9f35ef53c16bde9b54"} Oct 30 02:15:03.249034 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:03.249013 6221 scope.go:122] "RemoveContainer" containerID="c9b64aca6d2da8cdbd3d6c08797bcee0134167d9ded79a9f35ef53c16bde9b54" Oct 30 02:15:03.432489 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:03.432436 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888775-tvzg8" podStartSLOduration=3.432422882 podStartE2EDuration="3.432422882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:15:03.432105576 +0000 UTC m=+3427.010392912" watchObservedRunningTime="2026-10-30 02:15:03.432422882 +0000 UTC m=+3427.010710218" Oct 30 02:15:04.263211 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:04.263112 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-601d4b-29888775-d8mmg" event={"ID":"ebe12490-67c0-4d1d-ad75-1ade885d5650","Type":"ContainerStarted","Data":"0ba233234c5960a613f5f35d205dcbf17f479850d9c7de263ad26479f7682336"} Oct 30 02:15:04.265429 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:04.265400 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888775-gdp5b" event={"ID":"68d8d09d-2725-417b-8d9d-c0ea49ce29ac","Type":"ContainerStarted","Data":"1314217df8804745f016fa2d7f1ca279061e5b51f93baa89a94d25a61abc93a3"} Oct 30 02:15:04.267256 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:04.267232 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888775-mwqtg" event={"ID":"144c527d-0189-4c40-a64a-a207dfbdd710","Type":"ContainerStarted","Data":"4af24c08dd7b197e07edcacb8f8771b481a2b0ef7ff840bd84949a5a1cdaf496"} Oct 30 02:15:04.269289 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:04.269267 6221 generic.go:356] "Generic (PLEG): container finished" podID="c4ac5f29-bf87-42b2-82af-ee49cd16e879" containerID="5de1ee88759862400a5bd827020612f9896294dcd9118f9aa905ecc23b6dda23" exitCode=60 Oct 30 02:15:04.269369 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:04.269345 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888775-tvzg8" event={"ID":"c4ac5f29-bf87-42b2-82af-ee49cd16e879","Type":"ContainerDied","Data":"5de1ee88759862400a5bd827020612f9896294dcd9118f9aa905ecc23b6dda23"} Oct 30 02:15:04.269440 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:04.269402 6221 scope.go:122] "RemoveContainer" containerID="0dabc5db7f0ba80b2d1a5da7f7fa9d80ebf0400448a7acbb2d1f88a7a61216bd" Oct 30 02:15:04.269592 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:04.269570 6221 scope.go:122] "RemoveContainer" containerID="5de1ee88759862400a5bd827020612f9896294dcd9118f9aa905ecc23b6dda23" Oct 30 02:15:04.269910 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:15:04.269869 6221 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888775-tvzg8_odh-ai-gateway-infra(c4ac5f29-bf87-42b2-82af-ee49cd16e879)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888775-tvzg8" podUID="c4ac5f29-bf87-42b2-82af-ee49cd16e879" Oct 30 02:15:04.271930 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:04.271909 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-aaf045-29888775-zkr6d" event={"ID":"a506dae9-1ced-409d-99db-413b66dd9d59","Type":"ContainerStarted","Data":"1346a589886ef0986e1cb61e440d576e67fa72a38543cfa204e6ce42455569aa"} Oct 30 02:15:04.274045 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:04.274021 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888775-jrp5d" event={"ID":"66320d4a-0c8f-483d-98e4-b17658b6b27a","Type":"ContainerStarted","Data":"9a15d3e1ec54c5ddd231d833d71b02914af1f439c88315e917fb525fe2acce7f"} Oct 30 02:15:04.276272 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:04.276246 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22e99d-29888775-kbvl4" event={"ID":"d90c1ed4-5a28-4ce1-ba6e-c521698c3667","Type":"ContainerStarted","Data":"ae1c9c88b1e646e7b820d1bdac9f72296aefc3086d5c1976ac77392b018ba2de"} Oct 30 02:15:04.278330 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:04.278308 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888775-8zqzh" event={"ID":"def2849e-c94c-4685-9992-e1121b744ac8","Type":"ContainerStarted","Data":"e8c4bf6e85a819619feb19d49dd7c21de4e1835b148b23268e10ae9d0963a493"} Oct 30 02:15:04.280295 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:04.280275 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888775-x7l7p" event={"ID":"cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9","Type":"ContainerStarted","Data":"dc99fd177a052b55afec638dee9947f1ad116623e8208d2976a7c6ecf6cb4473"} Oct 30 02:15:05.291524 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:05.291488 6221 generic.go:356] "Generic (PLEG): container finished" podID="a506dae9-1ced-409d-99db-413b66dd9d59" containerID="1346a589886ef0986e1cb61e440d576e67fa72a38543cfa204e6ce42455569aa" exitCode=60 Oct 30 02:15:05.292051 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:05.291569 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-aaf045-29888775-zkr6d" event={"ID":"a506dae9-1ced-409d-99db-413b66dd9d59","Type":"ContainerDied","Data":"1346a589886ef0986e1cb61e440d576e67fa72a38543cfa204e6ce42455569aa"} Oct 30 02:15:05.292051 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:05.291635 6221 scope.go:122] "RemoveContainer" containerID="acd3b800d049fde420e28d06f0b127aa6d171b77c4cc484c0a8ed9271106bc3b" Oct 30 02:15:05.292051 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:05.291998 6221 scope.go:122] "RemoveContainer" containerID="1346a589886ef0986e1cb61e440d576e67fa72a38543cfa204e6ce42455569aa" Oct 30 02:15:05.292437 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:15:05.292415 6221 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-aaf045-29888775-zkr6d_odh-ai-gateway-infra(a506dae9-1ced-409d-99db-413b66dd9d59)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-aaf045-29888775-zkr6d" podUID="a506dae9-1ced-409d-99db-413b66dd9d59" Oct 30 02:15:05.293929 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:05.293907 6221 generic.go:356] "Generic (PLEG): container finished" podID="66320d4a-0c8f-483d-98e4-b17658b6b27a" containerID="9a15d3e1ec54c5ddd231d833d71b02914af1f439c88315e917fb525fe2acce7f" exitCode=60 Oct 30 02:15:05.294047 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:05.293984 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888775-jrp5d" event={"ID":"66320d4a-0c8f-483d-98e4-b17658b6b27a","Type":"ContainerDied","Data":"9a15d3e1ec54c5ddd231d833d71b02914af1f439c88315e917fb525fe2acce7f"} Oct 30 02:15:05.294316 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:05.294297 6221 scope.go:122] "RemoveContainer" containerID="9a15d3e1ec54c5ddd231d833d71b02914af1f439c88315e917fb525fe2acce7f" Oct 30 02:15:05.294728 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:15:05.294708 6221 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2e618f-29888775-jrp5d_odh-ai-gateway-infra(66320d4a-0c8f-483d-98e4-b17658b6b27a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888775-jrp5d" podUID="66320d4a-0c8f-483d-98e4-b17658b6b27a" Oct 30 02:15:05.296105 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:05.296084 6221 generic.go:356] "Generic (PLEG): container finished" podID="d90c1ed4-5a28-4ce1-ba6e-c521698c3667" containerID="ae1c9c88b1e646e7b820d1bdac9f72296aefc3086d5c1976ac77392b018ba2de" exitCode=60 Oct 30 02:15:05.296199 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:05.296119 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22e99d-29888775-kbvl4" event={"ID":"d90c1ed4-5a28-4ce1-ba6e-c521698c3667","Type":"ContainerDied","Data":"ae1c9c88b1e646e7b820d1bdac9f72296aefc3086d5c1976ac77392b018ba2de"} Oct 30 02:15:05.296469 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:05.296450 6221 scope.go:122] "RemoveContainer" containerID="ae1c9c88b1e646e7b820d1bdac9f72296aefc3086d5c1976ac77392b018ba2de" Oct 30 02:15:05.296806 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:15:05.296783 6221 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-22e99d-29888775-kbvl4_odh-ai-gateway-infra(d90c1ed4-5a28-4ce1-ba6e-c521698c3667)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22e99d-29888775-kbvl4" podUID="d90c1ed4-5a28-4ce1-ba6e-c521698c3667" Oct 30 02:15:05.298285 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:05.298265 6221 generic.go:356] "Generic (PLEG): container finished" podID="def2849e-c94c-4685-9992-e1121b744ac8" containerID="e8c4bf6e85a819619feb19d49dd7c21de4e1835b148b23268e10ae9d0963a493" exitCode=60 Oct 30 02:15:05.298371 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:05.298347 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888775-8zqzh" event={"ID":"def2849e-c94c-4685-9992-e1121b744ac8","Type":"ContainerDied","Data":"e8c4bf6e85a819619feb19d49dd7c21de4e1835b148b23268e10ae9d0963a493"} Oct 30 02:15:05.298654 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:05.298636 6221 scope.go:122] "RemoveContainer" containerID="e8c4bf6e85a819619feb19d49dd7c21de4e1835b148b23268e10ae9d0963a493" Oct 30 02:15:05.299003 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:15:05.298981 6221 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888775-8zqzh_odh-ai-gateway-infra(def2849e-c94c-4685-9992-e1121b744ac8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888775-8zqzh" podUID="def2849e-c94c-4685-9992-e1121b744ac8" Oct 30 02:15:05.300402 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:05.300384 6221 generic.go:356] "Generic (PLEG): container finished" podID="cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9" containerID="dc99fd177a052b55afec638dee9947f1ad116623e8208d2976a7c6ecf6cb4473" exitCode=60 Oct 30 02:15:05.300490 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:05.300457 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888775-x7l7p" event={"ID":"cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9","Type":"ContainerDied","Data":"dc99fd177a052b55afec638dee9947f1ad116623e8208d2976a7c6ecf6cb4473"} Oct 30 02:15:05.300735 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:05.300718 6221 scope.go:122] "RemoveContainer" containerID="dc99fd177a052b55afec638dee9947f1ad116623e8208d2976a7c6ecf6cb4473" Oct 30 02:15:05.301045 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:15:05.301021 6221 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-eda881-29888775-x7l7p_odh-ai-gateway-infra(cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888775-x7l7p" podUID="cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9" Oct 30 02:15:05.302384 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:05.302355 6221 generic.go:356] "Generic (PLEG): container finished" podID="ebe12490-67c0-4d1d-ad75-1ade885d5650" containerID="0ba233234c5960a613f5f35d205dcbf17f479850d9c7de263ad26479f7682336" exitCode=60 Oct 30 02:15:05.302483 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:05.302439 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-601d4b-29888775-d8mmg" event={"ID":"ebe12490-67c0-4d1d-ad75-1ade885d5650","Type":"ContainerDied","Data":"0ba233234c5960a613f5f35d205dcbf17f479850d9c7de263ad26479f7682336"} Oct 30 02:15:05.302698 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:05.302677 6221 scope.go:122] "RemoveContainer" containerID="0ba233234c5960a613f5f35d205dcbf17f479850d9c7de263ad26479f7682336" Oct 30 02:15:05.302960 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:15:05.302938 6221 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-601d4b-29888775-d8mmg_odh-ai-gateway-infra(ebe12490-67c0-4d1d-ad75-1ade885d5650)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-601d4b-29888775-d8mmg" podUID="ebe12490-67c0-4d1d-ad75-1ade885d5650" Oct 30 02:15:05.304408 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:05.304384 6221 generic.go:356] "Generic (PLEG): container finished" podID="68d8d09d-2725-417b-8d9d-c0ea49ce29ac" containerID="1314217df8804745f016fa2d7f1ca279061e5b51f93baa89a94d25a61abc93a3" exitCode=60 Oct 30 02:15:05.304408 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:05.304403 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888775-gdp5b" event={"ID":"68d8d09d-2725-417b-8d9d-c0ea49ce29ac","Type":"ContainerDied","Data":"1314217df8804745f016fa2d7f1ca279061e5b51f93baa89a94d25a61abc93a3"} Oct 30 02:15:05.304720 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:05.304701 6221 scope.go:122] "RemoveContainer" containerID="1314217df8804745f016fa2d7f1ca279061e5b51f93baa89a94d25a61abc93a3" Oct 30 02:15:05.304990 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:15:05.304972 6221 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888775-gdp5b_odh-ai-gateway-infra(68d8d09d-2725-417b-8d9d-c0ea49ce29ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888775-gdp5b" podUID="68d8d09d-2725-417b-8d9d-c0ea49ce29ac" Oct 30 02:15:05.306449 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:05.306423 6221 generic.go:356] "Generic (PLEG): container finished" podID="144c527d-0189-4c40-a64a-a207dfbdd710" containerID="4af24c08dd7b197e07edcacb8f8771b481a2b0ef7ff840bd84949a5a1cdaf496" exitCode=60 Oct 30 02:15:05.306692 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:05.306502 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888775-mwqtg" event={"ID":"144c527d-0189-4c40-a64a-a207dfbdd710","Type":"ContainerDied","Data":"4af24c08dd7b197e07edcacb8f8771b481a2b0ef7ff840bd84949a5a1cdaf496"} Oct 30 02:15:05.306692 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:05.306625 6221 scope.go:122] "RemoveContainer" containerID="3ebd3bede478a95c7d5840896c1f2bbf56e06d146f04b3699816e7d64d37f59f" Oct 30 02:15:05.306805 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:05.306708 6221 scope.go:122] "RemoveContainer" containerID="4af24c08dd7b197e07edcacb8f8771b481a2b0ef7ff840bd84949a5a1cdaf496" Oct 30 02:15:05.306998 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:15:05.306975 6221 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-76a19e-29888775-mwqtg_odh-ai-gateway-infra(144c527d-0189-4c40-a64a-a207dfbdd710)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888775-mwqtg" podUID="144c527d-0189-4c40-a64a-a207dfbdd710" Oct 30 02:15:05.308414 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:05.308398 6221 scope.go:122] "RemoveContainer" containerID="5de1ee88759862400a5bd827020612f9896294dcd9118f9aa905ecc23b6dda23" Oct 30 02:15:05.308633 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:15:05.308616 6221 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888775-tvzg8_odh-ai-gateway-infra(c4ac5f29-bf87-42b2-82af-ee49cd16e879)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888775-tvzg8" podUID="c4ac5f29-bf87-42b2-82af-ee49cd16e879" Oct 30 02:15:05.321109 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:05.321083 6221 scope.go:122] "RemoveContainer" containerID="c9b64aca6d2da8cdbd3d6c08797bcee0134167d9ded79a9f35ef53c16bde9b54" Oct 30 02:15:05.336265 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:05.336239 6221 scope.go:122] "RemoveContainer" containerID="6904aad63a54c111480fcca65ecef25893cb780f44a79009e79b9750ed20c5cc" Oct 30 02:15:05.350819 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:05.350796 6221 scope.go:122] "RemoveContainer" containerID="ee78fd27ea6af171734f00551887c2d5e6ec82929a149dcdd904fe1637ec0d1f" Oct 30 02:15:05.365301 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:05.365278 6221 scope.go:122] "RemoveContainer" containerID="21081a101abbc5aef195963a3397569ed035eeb582668f03fa60a031915ec78a" Oct 30 02:15:05.381743 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:05.381715 6221 scope.go:122] "RemoveContainer" containerID="f47cb1f76bbe425a0497bcedc239c6c382c9a4cb0ca37c393ddc806afa5fb6be" Oct 30 02:15:05.419493 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:05.419457 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr"] Oct 30 02:15:05.419931 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:05.419900 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" podUID="dcd89198-b4a1-40cd-9850-3dda3af7bbab" containerName="istio-proxy" containerID="cri-o://c81b77470d2ba487be1a1010d6da2c84db607c61e7ca0f6fdcd4f557cb1f022a" gracePeriod=30 Oct 30 02:15:05.429728 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:05.429608 6221 scope.go:122] "RemoveContainer" containerID="04ba3e18e0ea3f6ca5538a8c249fc98696a2e3d6c33d04ff8ff358a42c73e9c3" Oct 30 02:15:06.318239 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:06.318213 6221 scope.go:122] "RemoveContainer" containerID="1346a589886ef0986e1cb61e440d576e67fa72a38543cfa204e6ce42455569aa" Oct 30 02:15:06.318694 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:15:06.318629 6221 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-aaf045-29888775-zkr6d_odh-ai-gateway-infra(a506dae9-1ced-409d-99db-413b66dd9d59)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-aaf045-29888775-zkr6d" podUID="a506dae9-1ced-409d-99db-413b66dd9d59" Oct 30 02:15:06.319835 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:06.319816 6221 scope.go:122] "RemoveContainer" containerID="9a15d3e1ec54c5ddd231d833d71b02914af1f439c88315e917fb525fe2acce7f" Oct 30 02:15:06.320443 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:15:06.320169 6221 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2e618f-29888775-jrp5d_odh-ai-gateway-infra(66320d4a-0c8f-483d-98e4-b17658b6b27a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888775-jrp5d" podUID="66320d4a-0c8f-483d-98e4-b17658b6b27a" Oct 30 02:15:06.321581 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:06.321524 6221 scope.go:122] "RemoveContainer" containerID="ae1c9c88b1e646e7b820d1bdac9f72296aefc3086d5c1976ac77392b018ba2de" Oct 30 02:15:06.321889 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:15:06.321865 6221 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-22e99d-29888775-kbvl4_odh-ai-gateway-infra(d90c1ed4-5a28-4ce1-ba6e-c521698c3667)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22e99d-29888775-kbvl4" podUID="d90c1ed4-5a28-4ce1-ba6e-c521698c3667" Oct 30 02:15:06.323355 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:06.323333 6221 scope.go:122] "RemoveContainer" containerID="e8c4bf6e85a819619feb19d49dd7c21de4e1835b148b23268e10ae9d0963a493" Oct 30 02:15:06.323680 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:15:06.323651 6221 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888775-8zqzh_odh-ai-gateway-infra(def2849e-c94c-4685-9992-e1121b744ac8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888775-8zqzh" podUID="def2849e-c94c-4685-9992-e1121b744ac8" Oct 30 02:15:06.324930 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:06.324912 6221 scope.go:122] "RemoveContainer" containerID="dc99fd177a052b55afec638dee9947f1ad116623e8208d2976a7c6ecf6cb4473" Oct 30 02:15:06.325201 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:15:06.325181 6221 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-eda881-29888775-x7l7p_odh-ai-gateway-infra(cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888775-x7l7p" podUID="cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9" Oct 30 02:15:06.326711 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:06.326684 6221 scope.go:122] "RemoveContainer" containerID="0ba233234c5960a613f5f35d205dcbf17f479850d9c7de263ad26479f7682336" Oct 30 02:15:06.326969 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:15:06.326949 6221 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-601d4b-29888775-d8mmg_odh-ai-gateway-infra(ebe12490-67c0-4d1d-ad75-1ade885d5650)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-601d4b-29888775-d8mmg" podUID="ebe12490-67c0-4d1d-ad75-1ade885d5650" Oct 30 02:15:06.328400 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:06.328381 6221 scope.go:122] "RemoveContainer" containerID="1314217df8804745f016fa2d7f1ca279061e5b51f93baa89a94d25a61abc93a3" Oct 30 02:15:06.328674 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:15:06.328650 6221 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888775-gdp5b_odh-ai-gateway-infra(68d8d09d-2725-417b-8d9d-c0ea49ce29ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888775-gdp5b" podUID="68d8d09d-2725-417b-8d9d-c0ea49ce29ac" Oct 30 02:15:06.330353 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:06.330334 6221 scope.go:122] "RemoveContainer" containerID="4af24c08dd7b197e07edcacb8f8771b481a2b0ef7ff840bd84949a5a1cdaf496" Oct 30 02:15:06.330578 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:15:06.330560 6221 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-76a19e-29888775-mwqtg_odh-ai-gateway-infra(144c527d-0189-4c40-a64a-a207dfbdd710)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888775-mwqtg" podUID="144c527d-0189-4c40-a64a-a207dfbdd710" Oct 30 02:15:06.519926 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:06.519895 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4" Oct 30 02:15:06.608525 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:06.608492 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-istio-token\" (UniqueName: \"kubernetes.io/projected/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-istio-token\") pod \"de9bc101-0002-4af2-a1fa-ae9c6be2ff73\" (UID: \"de9bc101-0002-4af2-a1fa-ae9c6be2ff73\") " Oct 30 02:15:06.608715 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:06.608587 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-istiod-ca-cert\") pod \"de9bc101-0002-4af2-a1fa-ae9c6be2ff73\" (UID: \"de9bc101-0002-4af2-a1fa-ae9c6be2ff73\") " Oct 30 02:15:06.608715 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:06.608690 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-workload-socket\") pod \"de9bc101-0002-4af2-a1fa-ae9c6be2ff73\" (UID: \"de9bc101-0002-4af2-a1fa-ae9c6be2ff73\") " Oct 30 02:15:06.608833 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:06.608719 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-workload-certs\") pod \"de9bc101-0002-4af2-a1fa-ae9c6be2ff73\" (UID: \"de9bc101-0002-4af2-a1fa-ae9c6be2ff73\") " Oct 30 02:15:06.608833 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:06.608744 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-credential-socket\") pod \"de9bc101-0002-4af2-a1fa-ae9c6be2ff73\" (UID: \"de9bc101-0002-4af2-a1fa-ae9c6be2ff73\") " Oct 30 02:15:06.608833 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:06.608778 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-istio-podinfo\") pod \"de9bc101-0002-4af2-a1fa-ae9c6be2ff73\" (UID: \"de9bc101-0002-4af2-a1fa-ae9c6be2ff73\") " Oct 30 02:15:06.609001 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:06.608856 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-kube-api-access-kdrsb\" (UniqueName: \"kubernetes.io/projected/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-kube-api-access-kdrsb\") pod \"de9bc101-0002-4af2-a1fa-ae9c6be2ff73\" (UID: \"de9bc101-0002-4af2-a1fa-ae9c6be2ff73\") " Oct 30 02:15:06.609001 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:06.608892 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-istio-data\") pod \"de9bc101-0002-4af2-a1fa-ae9c6be2ff73\" (UID: \"de9bc101-0002-4af2-a1fa-ae9c6be2ff73\") " Oct 30 02:15:06.609001 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:06.608970 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-istio-envoy\") pod \"de9bc101-0002-4af2-a1fa-ae9c6be2ff73\" (UID: \"de9bc101-0002-4af2-a1fa-ae9c6be2ff73\") " Oct 30 02:15:06.609161 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:06.609004 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-workload-socket" pod "de9bc101-0002-4af2-a1fa-ae9c6be2ff73" (UID: "de9bc101-0002-4af2-a1fa-ae9c6be2ff73"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:15:06.609161 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:06.609051 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-workload-certs" pod "de9bc101-0002-4af2-a1fa-ae9c6be2ff73" (UID: "de9bc101-0002-4af2-a1fa-ae9c6be2ff73"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:15:06.609161 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:06.609062 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-credential-socket" pod "de9bc101-0002-4af2-a1fa-ae9c6be2ff73" (UID: "de9bc101-0002-4af2-a1fa-ae9c6be2ff73"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:15:06.609161 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:06.609100 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-istiod-ca-cert" pod "de9bc101-0002-4af2-a1fa-ae9c6be2ff73" (UID: "de9bc101-0002-4af2-a1fa-ae9c6be2ff73"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:06.609382 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:06.609317 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-istio-data" pod "de9bc101-0002-4af2-a1fa-ae9c6be2ff73" (UID: "de9bc101-0002-4af2-a1fa-ae9c6be2ff73"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:15:06.609782 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:06.609758 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:06.609885 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:06.609784 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:06.609885 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:06.609796 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:06.609885 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:06.609808 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:06.609885 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:06.609840 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:06.610929 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:06.610902 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-istio-token" pod "de9bc101-0002-4af2-a1fa-ae9c6be2ff73" (UID: "de9bc101-0002-4af2-a1fa-ae9c6be2ff73"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:06.611159 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:06.611142 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-kube-api-access-kdrsb" pod "de9bc101-0002-4af2-a1fa-ae9c6be2ff73" (UID: "de9bc101-0002-4af2-a1fa-ae9c6be2ff73"). InnerVolumeSpecName "kube-api-access-kdrsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:06.611580 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:06.611432 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-istio-envoy" pod "de9bc101-0002-4af2-a1fa-ae9c6be2ff73" (UID: "de9bc101-0002-4af2-a1fa-ae9c6be2ff73"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:15:06.611672 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:06.611653 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-istio-podinfo" pod "de9bc101-0002-4af2-a1fa-ae9c6be2ff73" (UID: "de9bc101-0002-4af2-a1fa-ae9c6be2ff73"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:15:06.711827 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:06.711791 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:06.711827 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:06.711824 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdrsb\" (UniqueName: \"kubernetes.io/projected/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-kube-api-access-kdrsb\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:06.711827 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:06.711837 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:06.712138 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:06.711846 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de9bc101-0002-4af2-a1fa-ae9c6be2ff73-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:07.339171 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:07.339131 6221 generic.go:356] "Generic (PLEG): container finished" podID="de9bc101-0002-4af2-a1fa-ae9c6be2ff73" containerID="7750b5c6e89c2dce8a4b1576b879fcdc33affd79f32bb3879266be3ac216d8d3" exitCode=0 Oct 30 02:15:07.339653 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:07.339199 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4" Oct 30 02:15:07.339653 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:07.339209 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4" event={"ID":"de9bc101-0002-4af2-a1fa-ae9c6be2ff73","Type":"ContainerDied","Data":"7750b5c6e89c2dce8a4b1576b879fcdc33affd79f32bb3879266be3ac216d8d3"} Oct 30 02:15:07.339653 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:07.339267 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4" event={"ID":"de9bc101-0002-4af2-a1fa-ae9c6be2ff73","Type":"ContainerDied","Data":"b643c12945f451573d713e0c65e15db94a9bdea03585adfa74e2cc91e29a3fc6"} Oct 30 02:15:07.339653 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:07.339289 6221 scope.go:122] "RemoveContainer" containerID="7750b5c6e89c2dce8a4b1576b879fcdc33affd79f32bb3879266be3ac216d8d3" Oct 30 02:15:07.352050 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:07.352029 6221 scope.go:122] "RemoveContainer" containerID="7750b5c6e89c2dce8a4b1576b879fcdc33affd79f32bb3879266be3ac216d8d3" Oct 30 02:15:07.352381 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:15:07.352360 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7750b5c6e89c2dce8a4b1576b879fcdc33affd79f32bb3879266be3ac216d8d3\": container with ID starting with 7750b5c6e89c2dce8a4b1576b879fcdc33affd79f32bb3879266be3ac216d8d3 not found: ID does not exist" containerID="7750b5c6e89c2dce8a4b1576b879fcdc33affd79f32bb3879266be3ac216d8d3" Oct 30 02:15:07.352456 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:07.352389 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7750b5c6e89c2dce8a4b1576b879fcdc33affd79f32bb3879266be3ac216d8d3"} err="failed to get container status \"7750b5c6e89c2dce8a4b1576b879fcdc33affd79f32bb3879266be3ac216d8d3\": rpc error: code = NotFound desc = could not find container \"7750b5c6e89c2dce8a4b1576b879fcdc33affd79f32bb3879266be3ac216d8d3\": container with ID starting with 7750b5c6e89c2dce8a4b1576b879fcdc33affd79f32bb3879266be3ac216d8d3 not found: ID does not exist" Oct 30 02:15:07.393739 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:07.393683 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4"] Oct 30 02:15:07.406151 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:07.406113 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-frjn4"] Oct 30 02:15:08.550671 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:08.550595 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de9bc101-0002-4af2-a1fa-ae9c6be2ff73" path="/var/lib/kubelet/pods/de9bc101-0002-4af2-a1fa-ae9c6be2ff73/volumes" Oct 30 02:15:11.105175 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:11.105142 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" Oct 30 02:15:11.191500 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:11.191415 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dcd89198-b4a1-40cd-9850-3dda3af7bbab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcd89198-b4a1-40cd-9850-3dda3af7bbab-istio-podinfo\") pod \"dcd89198-b4a1-40cd-9850-3dda3af7bbab\" (UID: \"dcd89198-b4a1-40cd-9850-3dda3af7bbab\") " Oct 30 02:15:11.191500 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:11.191484 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcd89198-b4a1-40cd-9850-3dda3af7bbab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcd89198-b4a1-40cd-9850-3dda3af7bbab-workload-certs\") pod \"dcd89198-b4a1-40cd-9850-3dda3af7bbab\" (UID: \"dcd89198-b4a1-40cd-9850-3dda3af7bbab\") " Oct 30 02:15:11.191748 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:11.191527 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcd89198-b4a1-40cd-9850-3dda3af7bbab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcd89198-b4a1-40cd-9850-3dda3af7bbab-istiod-ca-cert\") pod \"dcd89198-b4a1-40cd-9850-3dda3af7bbab\" (UID: \"dcd89198-b4a1-40cd-9850-3dda3af7bbab\") " Oct 30 02:15:11.191748 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:11.191599 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcd89198-b4a1-40cd-9850-3dda3af7bbab-istio-token\" (UniqueName: \"kubernetes.io/projected/dcd89198-b4a1-40cd-9850-3dda3af7bbab-istio-token\") pod \"dcd89198-b4a1-40cd-9850-3dda3af7bbab\" (UID: \"dcd89198-b4a1-40cd-9850-3dda3af7bbab\") " Oct 30 02:15:11.191748 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:11.191631 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcd89198-b4a1-40cd-9850-3dda3af7bbab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcd89198-b4a1-40cd-9850-3dda3af7bbab-istio-envoy\") pod \"dcd89198-b4a1-40cd-9850-3dda3af7bbab\" (UID: \"dcd89198-b4a1-40cd-9850-3dda3af7bbab\") " Oct 30 02:15:11.191748 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:11.191692 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcd89198-b4a1-40cd-9850-3dda3af7bbab-kube-api-access-9vzbx\" (UniqueName: \"kubernetes.io/projected/dcd89198-b4a1-40cd-9850-3dda3af7bbab-kube-api-access-9vzbx\") pod \"dcd89198-b4a1-40cd-9850-3dda3af7bbab\" (UID: \"dcd89198-b4a1-40cd-9850-3dda3af7bbab\") " Oct 30 02:15:11.191978 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:11.191776 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcd89198-b4a1-40cd-9850-3dda3af7bbab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcd89198-b4a1-40cd-9850-3dda3af7bbab-istio-data\") pod \"dcd89198-b4a1-40cd-9850-3dda3af7bbab\" (UID: \"dcd89198-b4a1-40cd-9850-3dda3af7bbab\") " Oct 30 02:15:11.191978 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:11.191874 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcd89198-b4a1-40cd-9850-3dda3af7bbab-workload-certs" pod "dcd89198-b4a1-40cd-9850-3dda3af7bbab" (UID: "dcd89198-b4a1-40cd-9850-3dda3af7bbab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:15:11.192082 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:11.191992 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcd89198-b4a1-40cd-9850-3dda3af7bbab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcd89198-b4a1-40cd-9850-3dda3af7bbab-credential-socket\") pod \"dcd89198-b4a1-40cd-9850-3dda3af7bbab\" (UID: \"dcd89198-b4a1-40cd-9850-3dda3af7bbab\") " Oct 30 02:15:11.192082 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:11.192018 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcd89198-b4a1-40cd-9850-3dda3af7bbab-istiod-ca-cert" pod "dcd89198-b4a1-40cd-9850-3dda3af7bbab" (UID: "dcd89198-b4a1-40cd-9850-3dda3af7bbab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:11.192082 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:11.192030 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcd89198-b4a1-40cd-9850-3dda3af7bbab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcd89198-b4a1-40cd-9850-3dda3af7bbab-workload-socket\") pod \"dcd89198-b4a1-40cd-9850-3dda3af7bbab\" (UID: \"dcd89198-b4a1-40cd-9850-3dda3af7bbab\") " Oct 30 02:15:11.192312 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:11.192290 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcd89198-b4a1-40cd-9850-3dda3af7bbab-workload-socket" pod "dcd89198-b4a1-40cd-9850-3dda3af7bbab" (UID: "dcd89198-b4a1-40cd-9850-3dda3af7bbab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:15:11.192707 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:11.192683 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcd89198-b4a1-40cd-9850-3dda3af7bbab-istio-data" pod "dcd89198-b4a1-40cd-9850-3dda3af7bbab" (UID: "dcd89198-b4a1-40cd-9850-3dda3af7bbab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:15:11.193057 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:11.193035 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcd89198-b4a1-40cd-9850-3dda3af7bbab-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:11.193205 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:11.193163 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcd89198-b4a1-40cd-9850-3dda3af7bbab-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:11.193321 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:11.193308 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcd89198-b4a1-40cd-9850-3dda3af7bbab-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:11.193429 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:11.193417 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcd89198-b4a1-40cd-9850-3dda3af7bbab-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:11.193525 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:11.193375 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcd89198-b4a1-40cd-9850-3dda3af7bbab-credential-socket" pod "dcd89198-b4a1-40cd-9850-3dda3af7bbab" (UID: "dcd89198-b4a1-40cd-9850-3dda3af7bbab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:15:11.194603 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:11.194515 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcd89198-b4a1-40cd-9850-3dda3af7bbab-istio-envoy" pod "dcd89198-b4a1-40cd-9850-3dda3af7bbab" (UID: "dcd89198-b4a1-40cd-9850-3dda3af7bbab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:15:11.194892 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:11.194864 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcd89198-b4a1-40cd-9850-3dda3af7bbab-istio-token" pod "dcd89198-b4a1-40cd-9850-3dda3af7bbab" (UID: "dcd89198-b4a1-40cd-9850-3dda3af7bbab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:11.195033 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:11.195013 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcd89198-b4a1-40cd-9850-3dda3af7bbab-kube-api-access-9vzbx" pod "dcd89198-b4a1-40cd-9850-3dda3af7bbab" (UID: "dcd89198-b4a1-40cd-9850-3dda3af7bbab"). InnerVolumeSpecName "kube-api-access-9vzbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:11.195033 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:11.195023 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcd89198-b4a1-40cd-9850-3dda3af7bbab-istio-podinfo" pod "dcd89198-b4a1-40cd-9850-3dda3af7bbab" (UID: "dcd89198-b4a1-40cd-9850-3dda3af7bbab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:15:11.294990 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:11.294951 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcd89198-b4a1-40cd-9850-3dda3af7bbab-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:11.294990 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:11.294984 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcd89198-b4a1-40cd-9850-3dda3af7bbab-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:11.294990 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:11.294999 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcd89198-b4a1-40cd-9850-3dda3af7bbab-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:11.295245 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:11.295010 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcd89198-b4a1-40cd-9850-3dda3af7bbab-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:11.295245 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:11.295021 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vzbx\" (UniqueName: \"kubernetes.io/projected/dcd89198-b4a1-40cd-9850-3dda3af7bbab-kube-api-access-9vzbx\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:11.376541 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:11.376486 6221 generic.go:356] "Generic (PLEG): container finished" podID="dcd89198-b4a1-40cd-9850-3dda3af7bbab" containerID="c81b77470d2ba487be1a1010d6da2c84db607c61e7ca0f6fdcd4f557cb1f022a" exitCode=0 Oct 30 02:15:11.376723 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:11.376555 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" Oct 30 02:15:11.376723 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:11.376573 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" event={"ID":"dcd89198-b4a1-40cd-9850-3dda3af7bbab","Type":"ContainerDied","Data":"c81b77470d2ba487be1a1010d6da2c84db607c61e7ca0f6fdcd4f557cb1f022a"} Oct 30 02:15:11.376723 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:11.376630 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr" event={"ID":"dcd89198-b4a1-40cd-9850-3dda3af7bbab","Type":"ContainerDied","Data":"47da10f24f2938ec92cbc9e3b732dccf31d16773aea16c5aefca012f51a2023b"} Oct 30 02:15:11.376723 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:11.376654 6221 scope.go:122] "RemoveContainer" containerID="c81b77470d2ba487be1a1010d6da2c84db607c61e7ca0f6fdcd4f557cb1f022a" Oct 30 02:15:11.389593 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:11.389571 6221 scope.go:122] "RemoveContainer" containerID="aa3a19507a0829c2b275082658f45f955b9aa27e73a1190ae70ece556cb20d8c" Oct 30 02:15:11.400737 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:11.400712 6221 scope.go:122] "RemoveContainer" containerID="c81b77470d2ba487be1a1010d6da2c84db607c61e7ca0f6fdcd4f557cb1f022a" Oct 30 02:15:11.401069 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:15:11.401046 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c81b77470d2ba487be1a1010d6da2c84db607c61e7ca0f6fdcd4f557cb1f022a\": container with ID starting with c81b77470d2ba487be1a1010d6da2c84db607c61e7ca0f6fdcd4f557cb1f022a not found: ID does not exist" containerID="c81b77470d2ba487be1a1010d6da2c84db607c61e7ca0f6fdcd4f557cb1f022a" Oct 30 02:15:11.401144 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:11.401078 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c81b77470d2ba487be1a1010d6da2c84db607c61e7ca0f6fdcd4f557cb1f022a"} err="failed to get container status \"c81b77470d2ba487be1a1010d6da2c84db607c61e7ca0f6fdcd4f557cb1f022a\": rpc error: code = NotFound desc = could not find container \"c81b77470d2ba487be1a1010d6da2c84db607c61e7ca0f6fdcd4f557cb1f022a\": container with ID starting with c81b77470d2ba487be1a1010d6da2c84db607c61e7ca0f6fdcd4f557cb1f022a not found: ID does not exist" Oct 30 02:15:11.401144 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:11.401097 6221 scope.go:122] "RemoveContainer" containerID="aa3a19507a0829c2b275082658f45f955b9aa27e73a1190ae70ece556cb20d8c" Oct 30 02:15:11.401321 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:15:11.401304 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa3a19507a0829c2b275082658f45f955b9aa27e73a1190ae70ece556cb20d8c\": container with ID starting with aa3a19507a0829c2b275082658f45f955b9aa27e73a1190ae70ece556cb20d8c not found: ID does not exist" containerID="aa3a19507a0829c2b275082658f45f955b9aa27e73a1190ae70ece556cb20d8c" Oct 30 02:15:11.401366 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:11.401327 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa3a19507a0829c2b275082658f45f955b9aa27e73a1190ae70ece556cb20d8c"} err="failed to get container status \"aa3a19507a0829c2b275082658f45f955b9aa27e73a1190ae70ece556cb20d8c\": rpc error: code = NotFound desc = could not find container \"aa3a19507a0829c2b275082658f45f955b9aa27e73a1190ae70ece556cb20d8c\": container with ID starting with aa3a19507a0829c2b275082658f45f955b9aa27e73a1190ae70ece556cb20d8c not found: ID does not exist" Oct 30 02:15:11.430725 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:11.430696 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr"] Oct 30 02:15:11.444923 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:11.444848 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-2eae39-openshift-default-56fd9fd645-fvllr"] Oct 30 02:15:12.550083 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:12.550050 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcd89198-b4a1-40cd-9850-3dda3af7bbab" path="/var/lib/kubelet/pods/dcd89198-b4a1-40cd-9850-3dda3af7bbab/volumes" Oct 30 02:15:15.540630 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:15.540600 6221 scope.go:122] "RemoveContainer" containerID="5de1ee88759862400a5bd827020612f9896294dcd9118f9aa905ecc23b6dda23" Oct 30 02:15:15.541104 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:15.540754 6221 scope.go:122] "RemoveContainer" containerID="1314217df8804745f016fa2d7f1ca279061e5b51f93baa89a94d25a61abc93a3" Oct 30 02:15:15.541104 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:15.541021 6221 scope.go:122] "RemoveContainer" containerID="1346a589886ef0986e1cb61e440d576e67fa72a38543cfa204e6ce42455569aa" Oct 30 02:15:16.416917 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:16.416882 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-cd8173-5674dfcc9f-sqswj"] Oct 30 02:15:16.417951 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:16.417914 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888775-gdp5b" event={"ID":"68d8d09d-2725-417b-8d9d-c0ea49ce29ac","Type":"ContainerStarted","Data":"2cdeff6b11bee4f93d3b6f426f3ae9b74baebb6121e20b3e34b1200f2ada1096"} Oct 30 02:15:16.420773 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:16.420747 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888775-tvzg8" event={"ID":"c4ac5f29-bf87-42b2-82af-ee49cd16e879","Type":"ContainerStarted","Data":"24d73b91c12f6f2a26148f9b285cd8b1891d372dcf43fcf7c2fd625225d70ff2"} Oct 30 02:15:16.423034 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:16.423010 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-aaf045-29888775-zkr6d" event={"ID":"a506dae9-1ced-409d-99db-413b66dd9d59","Type":"ContainerStarted","Data":"2d1cd72c4576fa8d4ba6da134a35815775ce39026233b74c1f79d5a5ae6ae830"} Oct 30 02:15:16.540236 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:16.540184 6221 scope.go:122] "RemoveContainer" containerID="dc99fd177a052b55afec638dee9947f1ad116623e8208d2976a7c6ecf6cb4473" Oct 30 02:15:16.540408 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:16.540315 6221 scope.go:122] "RemoveContainer" containerID="9a15d3e1ec54c5ddd231d833d71b02914af1f439c88315e917fb525fe2acce7f" Oct 30 02:15:16.541945 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:16.541918 6221 scope.go:122] "RemoveContainer" containerID="ae1c9c88b1e646e7b820d1bdac9f72296aefc3086d5c1976ac77392b018ba2de" Oct 30 02:15:17.438553 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:17.438433 6221 generic.go:356] "Generic (PLEG): container finished" podID="a506dae9-1ced-409d-99db-413b66dd9d59" containerID="2d1cd72c4576fa8d4ba6da134a35815775ce39026233b74c1f79d5a5ae6ae830" exitCode=60 Oct 30 02:15:17.438769 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:17.438510 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-aaf045-29888775-zkr6d" event={"ID":"a506dae9-1ced-409d-99db-413b66dd9d59","Type":"ContainerDied","Data":"2d1cd72c4576fa8d4ba6da134a35815775ce39026233b74c1f79d5a5ae6ae830"} Oct 30 02:15:17.438769 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:17.438615 6221 scope.go:122] "RemoveContainer" containerID="1346a589886ef0986e1cb61e440d576e67fa72a38543cfa204e6ce42455569aa" Oct 30 02:15:17.439479 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:17.439455 6221 scope.go:122] "RemoveContainer" containerID="2d1cd72c4576fa8d4ba6da134a35815775ce39026233b74c1f79d5a5ae6ae830" Oct 30 02:15:17.440149 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:15:17.439892 6221 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-aaf045-29888775-zkr6d_odh-ai-gateway-infra(a506dae9-1ced-409d-99db-413b66dd9d59)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-aaf045-29888775-zkr6d" podUID="a506dae9-1ced-409d-99db-413b66dd9d59" Oct 30 02:15:17.443591 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:17.443567 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888775-jrp5d" event={"ID":"66320d4a-0c8f-483d-98e4-b17658b6b27a","Type":"ContainerStarted","Data":"ec49438ab901962861e75a9f970a4adef0bce5a1d54fb5e216aa940642dff642"} Oct 30 02:15:17.446116 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:17.446092 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22e99d-29888775-kbvl4" event={"ID":"d90c1ed4-5a28-4ce1-ba6e-c521698c3667","Type":"ContainerStarted","Data":"ec8cce93a44657c3413340b86621165538e9272585af5dff22b6677d95e3e58e"} Oct 30 02:15:17.448985 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:17.448959 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888775-x7l7p" event={"ID":"cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9","Type":"ContainerStarted","Data":"1c1bcc4b8e6c97310cf7d0ee2751730ba04f81f1354d9e805aa1721e0ef25949"} Oct 30 02:15:17.451733 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:17.451713 6221 generic.go:356] "Generic (PLEG): container finished" podID="68d8d09d-2725-417b-8d9d-c0ea49ce29ac" containerID="2cdeff6b11bee4f93d3b6f426f3ae9b74baebb6121e20b3e34b1200f2ada1096" exitCode=60 Oct 30 02:15:17.451834 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:17.451787 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888775-gdp5b" event={"ID":"68d8d09d-2725-417b-8d9d-c0ea49ce29ac","Type":"ContainerDied","Data":"2cdeff6b11bee4f93d3b6f426f3ae9b74baebb6121e20b3e34b1200f2ada1096"} Oct 30 02:15:17.452091 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:17.452077 6221 scope.go:122] "RemoveContainer" containerID="2cdeff6b11bee4f93d3b6f426f3ae9b74baebb6121e20b3e34b1200f2ada1096" Oct 30 02:15:17.452346 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:15:17.452328 6221 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888775-gdp5b_odh-ai-gateway-infra(68d8d09d-2725-417b-8d9d-c0ea49ce29ac)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888775-gdp5b" podUID="68d8d09d-2725-417b-8d9d-c0ea49ce29ac" Oct 30 02:15:17.454799 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:17.454779 6221 generic.go:356] "Generic (PLEG): container finished" podID="c4ac5f29-bf87-42b2-82af-ee49cd16e879" containerID="24d73b91c12f6f2a26148f9b285cd8b1891d372dcf43fcf7c2fd625225d70ff2" exitCode=60 Oct 30 02:15:17.454914 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:17.454852 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888775-tvzg8" event={"ID":"c4ac5f29-bf87-42b2-82af-ee49cd16e879","Type":"ContainerDied","Data":"24d73b91c12f6f2a26148f9b285cd8b1891d372dcf43fcf7c2fd625225d70ff2"} Oct 30 02:15:17.455158 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:17.455141 6221 scope.go:122] "RemoveContainer" containerID="24d73b91c12f6f2a26148f9b285cd8b1891d372dcf43fcf7c2fd625225d70ff2" Oct 30 02:15:17.455429 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:15:17.455412 6221 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888775-tvzg8_odh-ai-gateway-infra(c4ac5f29-bf87-42b2-82af-ee49cd16e879)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888775-tvzg8" podUID="c4ac5f29-bf87-42b2-82af-ee49cd16e879" Oct 30 02:15:17.457078 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:17.457059 6221 scope.go:122] "RemoveContainer" containerID="1314217df8804745f016fa2d7f1ca279061e5b51f93baa89a94d25a61abc93a3" Oct 30 02:15:17.471208 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:17.471184 6221 scope.go:122] "RemoveContainer" containerID="5de1ee88759862400a5bd827020612f9896294dcd9118f9aa905ecc23b6dda23" Oct 30 02:15:17.473042 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:17.473023 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 02:15:17.504825 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:17.504798 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888775-gdp5b"] Oct 30 02:15:17.552072 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:17.552035 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888775-tvzg8"] Oct 30 02:15:17.597683 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:17.597644 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-aaf045-29888775-zkr6d"] Oct 30 02:15:18.477801 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:18.477757 6221 generic.go:356] "Generic (PLEG): container finished" podID="66320d4a-0c8f-483d-98e4-b17658b6b27a" containerID="ec49438ab901962861e75a9f970a4adef0bce5a1d54fb5e216aa940642dff642" exitCode=60 Oct 30 02:15:18.477931 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:18.477886 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888775-jrp5d" event={"ID":"66320d4a-0c8f-483d-98e4-b17658b6b27a","Type":"ContainerDied","Data":"ec49438ab901962861e75a9f970a4adef0bce5a1d54fb5e216aa940642dff642"} Oct 30 02:15:18.477990 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:18.477944 6221 scope.go:122] "RemoveContainer" containerID="9a15d3e1ec54c5ddd231d833d71b02914af1f439c88315e917fb525fe2acce7f" Oct 30 02:15:18.478543 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:18.478507 6221 scope.go:122] "RemoveContainer" containerID="ec49438ab901962861e75a9f970a4adef0bce5a1d54fb5e216aa940642dff642" Oct 30 02:15:18.479108 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:15:18.479074 6221 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2e618f-29888775-jrp5d_odh-ai-gateway-infra(66320d4a-0c8f-483d-98e4-b17658b6b27a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888775-jrp5d" podUID="66320d4a-0c8f-483d-98e4-b17658b6b27a" Oct 30 02:15:18.481332 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:18.481302 6221 generic.go:356] "Generic (PLEG): container finished" podID="d90c1ed4-5a28-4ce1-ba6e-c521698c3667" containerID="ec8cce93a44657c3413340b86621165538e9272585af5dff22b6677d95e3e58e" exitCode=60 Oct 30 02:15:18.481417 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:18.481380 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22e99d-29888775-kbvl4" event={"ID":"d90c1ed4-5a28-4ce1-ba6e-c521698c3667","Type":"ContainerDied","Data":"ec8cce93a44657c3413340b86621165538e9272585af5dff22b6677d95e3e58e"} Oct 30 02:15:18.482016 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:18.481845 6221 scope.go:122] "RemoveContainer" containerID="ec8cce93a44657c3413340b86621165538e9272585af5dff22b6677d95e3e58e" Oct 30 02:15:18.482288 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:15:18.482228 6221 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-22e99d-29888775-kbvl4_odh-ai-gateway-infra(d90c1ed4-5a28-4ce1-ba6e-c521698c3667)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22e99d-29888775-kbvl4" podUID="d90c1ed4-5a28-4ce1-ba6e-c521698c3667" Oct 30 02:15:18.484898 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:18.484865 6221 generic.go:356] "Generic (PLEG): container finished" podID="cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9" containerID="1c1bcc4b8e6c97310cf7d0ee2751730ba04f81f1354d9e805aa1721e0ef25949" exitCode=60 Oct 30 02:15:18.485633 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:18.484996 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888775-x7l7p" event={"ID":"cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9","Type":"ContainerDied","Data":"1c1bcc4b8e6c97310cf7d0ee2751730ba04f81f1354d9e805aa1721e0ef25949"} Oct 30 02:15:18.485633 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:18.485314 6221 scope.go:122] "RemoveContainer" containerID="1c1bcc4b8e6c97310cf7d0ee2751730ba04f81f1354d9e805aa1721e0ef25949" Oct 30 02:15:18.485890 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:15:18.485846 6221 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-eda881-29888775-x7l7p_odh-ai-gateway-infra(cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888775-x7l7p" podUID="cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9" Oct 30 02:15:18.495734 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:18.495599 6221 scope.go:122] "RemoveContainer" containerID="ae1c9c88b1e646e7b820d1bdac9f72296aefc3086d5c1976ac77392b018ba2de" Oct 30 02:15:18.531387 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:18.530931 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888775-x7l7p"] Oct 30 02:15:18.556130 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:18.556103 6221 scope.go:122] "RemoveContainer" containerID="dc99fd177a052b55afec638dee9947f1ad116623e8208d2976a7c6ecf6cb4473" Oct 30 02:15:18.557225 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:18.556751 6221 scope.go:122] "RemoveContainer" containerID="0ba233234c5960a613f5f35d205dcbf17f479850d9c7de263ad26479f7682336" Oct 30 02:15:18.557225 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:18.557189 6221 scope.go:122] "RemoveContainer" containerID="4af24c08dd7b197e07edcacb8f8771b481a2b0ef7ff840bd84949a5a1cdaf496" Oct 30 02:15:18.620935 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:18.620901 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888775-jrp5d"] Oct 30 02:15:18.987467 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:18.977518 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22e99d-29888775-kbvl4"] Oct 30 02:15:19.048090 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.047713 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-aaf045-29888775-zkr6d" Oct 30 02:15:19.104372 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.104343 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888775-tvzg8" Oct 30 02:15:19.113152 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.113054 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888775-gdp5b" Oct 30 02:15:19.177904 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.177865 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a506dae9-1ced-409d-99db-413b66dd9d59-kube-api-access-4vbqt\" (UniqueName: \"kubernetes.io/projected/a506dae9-1ced-409d-99db-413b66dd9d59-kube-api-access-4vbqt\") pod \"a506dae9-1ced-409d-99db-413b66dd9d59\" (UID: \"a506dae9-1ced-409d-99db-413b66dd9d59\") " Oct 30 02:15:19.177904 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.177917 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a506dae9-1ced-409d-99db-413b66dd9d59-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a506dae9-1ced-409d-99db-413b66dd9d59-maas-api-service-ca\") pod \"a506dae9-1ced-409d-99db-413b66dd9d59\" (UID: \"a506dae9-1ced-409d-99db-413b66dd9d59\") " Oct 30 02:15:19.178695 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.178664 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a506dae9-1ced-409d-99db-413b66dd9d59-maas-api-service-ca" pod "a506dae9-1ced-409d-99db-413b66dd9d59" (UID: "a506dae9-1ced-409d-99db-413b66dd9d59"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:19.208610 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.208573 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a506dae9-1ced-409d-99db-413b66dd9d59-kube-api-access-4vbqt" pod "a506dae9-1ced-409d-99db-413b66dd9d59" (UID: "a506dae9-1ced-409d-99db-413b66dd9d59"). InnerVolumeSpecName "kube-api-access-4vbqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:19.279724 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.279692 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4ac5f29-bf87-42b2-82af-ee49cd16e879-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4ac5f29-bf87-42b2-82af-ee49cd16e879-maas-api-service-ca\") pod \"c4ac5f29-bf87-42b2-82af-ee49cd16e879\" (UID: \"c4ac5f29-bf87-42b2-82af-ee49cd16e879\") " Oct 30 02:15:19.279924 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.279803 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4ac5f29-bf87-42b2-82af-ee49cd16e879-kube-api-access-z4st2\" (UniqueName: \"kubernetes.io/projected/c4ac5f29-bf87-42b2-82af-ee49cd16e879-kube-api-access-z4st2\") pod \"c4ac5f29-bf87-42b2-82af-ee49cd16e879\" (UID: \"c4ac5f29-bf87-42b2-82af-ee49cd16e879\") " Oct 30 02:15:19.280034 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.280019 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68d8d09d-2725-417b-8d9d-c0ea49ce29ac-kube-api-access-795f8\" (UniqueName: \"kubernetes.io/projected/68d8d09d-2725-417b-8d9d-c0ea49ce29ac-kube-api-access-795f8\") pod \"68d8d09d-2725-417b-8d9d-c0ea49ce29ac\" (UID: \"68d8d09d-2725-417b-8d9d-c0ea49ce29ac\") " Oct 30 02:15:19.280144 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.280089 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68d8d09d-2725-417b-8d9d-c0ea49ce29ac-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68d8d09d-2725-417b-8d9d-c0ea49ce29ac-maas-api-service-ca\") pod \"68d8d09d-2725-417b-8d9d-c0ea49ce29ac\" (UID: \"68d8d09d-2725-417b-8d9d-c0ea49ce29ac\") " Oct 30 02:15:19.280206 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.280149 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4ac5f29-bf87-42b2-82af-ee49cd16e879-maas-api-service-ca" pod "c4ac5f29-bf87-42b2-82af-ee49cd16e879" (UID: "c4ac5f29-bf87-42b2-82af-ee49cd16e879"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:19.280690 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.280666 6221 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4ac5f29-bf87-42b2-82af-ee49cd16e879-maas-api-service-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:19.280808 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.280695 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vbqt\" (UniqueName: \"kubernetes.io/projected/a506dae9-1ced-409d-99db-413b66dd9d59-kube-api-access-4vbqt\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:19.280808 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.280710 6221 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a506dae9-1ced-409d-99db-413b66dd9d59-maas-api-service-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:19.281020 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.280995 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68d8d09d-2725-417b-8d9d-c0ea49ce29ac-maas-api-service-ca" pod "68d8d09d-2725-417b-8d9d-c0ea49ce29ac" (UID: "68d8d09d-2725-417b-8d9d-c0ea49ce29ac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:19.282556 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.282513 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4ac5f29-bf87-42b2-82af-ee49cd16e879-kube-api-access-z4st2" pod "c4ac5f29-bf87-42b2-82af-ee49cd16e879" (UID: "c4ac5f29-bf87-42b2-82af-ee49cd16e879"). InnerVolumeSpecName "kube-api-access-z4st2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:19.283039 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.283008 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68d8d09d-2725-417b-8d9d-c0ea49ce29ac-kube-api-access-795f8" pod "68d8d09d-2725-417b-8d9d-c0ea49ce29ac" (UID: "68d8d09d-2725-417b-8d9d-c0ea49ce29ac"). InnerVolumeSpecName "kube-api-access-795f8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:19.382550 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.382448 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4st2\" (UniqueName: \"kubernetes.io/projected/c4ac5f29-bf87-42b2-82af-ee49cd16e879-kube-api-access-z4st2\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:19.382550 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.382479 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-795f8\" (UniqueName: \"kubernetes.io/projected/68d8d09d-2725-417b-8d9d-c0ea49ce29ac-kube-api-access-795f8\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:19.382550 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.382490 6221 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68d8d09d-2725-417b-8d9d-c0ea49ce29ac-maas-api-service-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:19.499419 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.499382 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-601d4b-29888775-d8mmg" event={"ID":"ebe12490-67c0-4d1d-ad75-1ade885d5650","Type":"ContainerStarted","Data":"09d17e83cfccba3109c2babe5683f3284cef7471a3334bcbe50b4600f27c9391"} Oct 30 02:15:19.500859 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.500835 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888775-gdp5b" Oct 30 02:15:19.500859 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.500848 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888775-gdp5b" event={"ID":"68d8d09d-2725-417b-8d9d-c0ea49ce29ac","Type":"ContainerDied","Data":"246e07c750089299f50e207133ead594a4d9e2d424459248eef3f287d3641f1c"} Oct 30 02:15:19.501070 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.500903 6221 scope.go:122] "RemoveContainer" containerID="2cdeff6b11bee4f93d3b6f426f3ae9b74baebb6121e20b3e34b1200f2ada1096" Oct 30 02:15:19.503298 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.503276 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888775-mwqtg" event={"ID":"144c527d-0189-4c40-a64a-a207dfbdd710","Type":"ContainerStarted","Data":"3b863fa5e4ad760c3d169f15e5690690434ad29752da105c53373175f5819e40"} Oct 30 02:15:19.505005 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.504977 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888775-tvzg8" event={"ID":"c4ac5f29-bf87-42b2-82af-ee49cd16e879","Type":"ContainerDied","Data":"9930c68c5c436977368c934870754a40f2f58cd88502653487fbf5625d517273"} Oct 30 02:15:19.505107 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.505044 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888775-tvzg8" Oct 30 02:15:19.506917 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.506892 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-aaf045-29888775-zkr6d" event={"ID":"a506dae9-1ced-409d-99db-413b66dd9d59","Type":"ContainerDied","Data":"d13a85c746e9976b87e7f70055baf4f5cbfcbc58b79b1a8fd1ff4f054820ab83"} Oct 30 02:15:19.507012 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.506989 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-aaf045-29888775-zkr6d" Oct 30 02:15:19.519688 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.519622 6221 scope.go:122] "RemoveContainer" containerID="24d73b91c12f6f2a26148f9b285cd8b1891d372dcf43fcf7c2fd625225d70ff2" Oct 30 02:15:19.537705 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.537339 6221 scope.go:122] "RemoveContainer" containerID="2d1cd72c4576fa8d4ba6da134a35815775ce39026233b74c1f79d5a5ae6ae830" Oct 30 02:15:19.541957 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.541450 6221 scope.go:122] "RemoveContainer" containerID="e8c4bf6e85a819619feb19d49dd7c21de4e1835b148b23268e10ae9d0963a493" Oct 30 02:15:19.644661 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.644602 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-aaf045-29888775-zkr6d"] Oct 30 02:15:19.649361 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.649332 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-aaf045-29888775-zkr6d"] Oct 30 02:15:19.773334 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.773303 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888775-jrp5d" Oct 30 02:15:19.839061 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.838893 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888775-gdp5b"] Oct 30 02:15:19.857353 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.857318 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-cf82f8-29888775-gdp5b"] Oct 30 02:15:19.872906 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.872883 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888775-x7l7p" Oct 30 02:15:19.876857 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.876836 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22e99d-29888775-kbvl4" Oct 30 02:15:19.894410 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.894348 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66320d4a-0c8f-483d-98e4-b17658b6b27a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66320d4a-0c8f-483d-98e4-b17658b6b27a-maas-api-service-ca\") pod \"66320d4a-0c8f-483d-98e4-b17658b6b27a\" (UID: \"66320d4a-0c8f-483d-98e4-b17658b6b27a\") " Oct 30 02:15:19.894520 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.894416 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66320d4a-0c8f-483d-98e4-b17658b6b27a-kube-api-access-mdxjg\" (UniqueName: \"kubernetes.io/projected/66320d4a-0c8f-483d-98e4-b17658b6b27a-kube-api-access-mdxjg\") pod \"66320d4a-0c8f-483d-98e4-b17658b6b27a\" (UID: \"66320d4a-0c8f-483d-98e4-b17658b6b27a\") " Oct 30 02:15:19.894652 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.894632 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66320d4a-0c8f-483d-98e4-b17658b6b27a-maas-api-service-ca" pod "66320d4a-0c8f-483d-98e4-b17658b6b27a" (UID: "66320d4a-0c8f-483d-98e4-b17658b6b27a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:19.894972 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.894952 6221 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66320d4a-0c8f-483d-98e4-b17658b6b27a-maas-api-service-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:19.896579 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.896551 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66320d4a-0c8f-483d-98e4-b17658b6b27a-kube-api-access-mdxjg" pod "66320d4a-0c8f-483d-98e4-b17658b6b27a" (UID: "66320d4a-0c8f-483d-98e4-b17658b6b27a"). InnerVolumeSpecName "kube-api-access-mdxjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:19.916459 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.916430 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888775-tvzg8"] Oct 30 02:15:19.933638 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.933609 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-861cd2-29888775-tvzg8"] Oct 30 02:15:19.996128 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.996076 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d90c1ed4-5a28-4ce1-ba6e-c521698c3667-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d90c1ed4-5a28-4ce1-ba6e-c521698c3667-maas-api-service-ca\") pod \"d90c1ed4-5a28-4ce1-ba6e-c521698c3667\" (UID: \"d90c1ed4-5a28-4ce1-ba6e-c521698c3667\") " Oct 30 02:15:19.996290 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.996227 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9-maas-api-service-ca\") pod \"cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9\" (UID: \"cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9\") " Oct 30 02:15:19.996433 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.996412 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d90c1ed4-5a28-4ce1-ba6e-c521698c3667-kube-api-access-nnsdb\" (UniqueName: \"kubernetes.io/projected/d90c1ed4-5a28-4ce1-ba6e-c521698c3667-kube-api-access-nnsdb\") pod \"d90c1ed4-5a28-4ce1-ba6e-c521698c3667\" (UID: \"d90c1ed4-5a28-4ce1-ba6e-c521698c3667\") " Oct 30 02:15:19.996558 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.996458 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9-kube-api-access-jvl5m\" (UniqueName: \"kubernetes.io/projected/cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9-kube-api-access-jvl5m\") pod \"cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9\" (UID: \"cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9\") " Oct 30 02:15:19.996558 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.996500 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d90c1ed4-5a28-4ce1-ba6e-c521698c3667-maas-api-service-ca" pod "d90c1ed4-5a28-4ce1-ba6e-c521698c3667" (UID: "d90c1ed4-5a28-4ce1-ba6e-c521698c3667"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:19.996648 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.996582 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9-maas-api-service-ca" pod "cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9" (UID: "cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:19.997012 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.996980 6221 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d90c1ed4-5a28-4ce1-ba6e-c521698c3667-maas-api-service-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:19.997012 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.997000 6221 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9-maas-api-service-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:19.997012 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.997011 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdxjg\" (UniqueName: \"kubernetes.io/projected/66320d4a-0c8f-483d-98e4-b17658b6b27a-kube-api-access-mdxjg\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:19.999032 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.999006 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d90c1ed4-5a28-4ce1-ba6e-c521698c3667-kube-api-access-nnsdb" pod "d90c1ed4-5a28-4ce1-ba6e-c521698c3667" (UID: "d90c1ed4-5a28-4ce1-ba6e-c521698c3667"). InnerVolumeSpecName "kube-api-access-nnsdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:19.999121 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:19.999046 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9-kube-api-access-jvl5m" pod "cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9" (UID: "cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9"). InnerVolumeSpecName "kube-api-access-jvl5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:20.098506 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:20.098464 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnsdb\" (UniqueName: \"kubernetes.io/projected/d90c1ed4-5a28-4ce1-ba6e-c521698c3667-kube-api-access-nnsdb\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:20.098506 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:20.098502 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvl5m\" (UniqueName: \"kubernetes.io/projected/cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9-kube-api-access-jvl5m\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:20.535353 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:20.535319 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888775-8zqzh" event={"ID":"def2849e-c94c-4685-9992-e1121b744ac8","Type":"ContainerStarted","Data":"355c5b91826c434e36396766ecd25141bbb931f9a1ae495f9b0ff6ddc5603c45"} Oct 30 02:15:20.536517 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:20.536497 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888775-x7l7p" Oct 30 02:15:20.536707 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:20.536554 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888775-x7l7p" event={"ID":"cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9","Type":"ContainerDied","Data":"19d8fed769322b7c48c82dd8376f5a64ee5ec60342f25f9de9f27e49acfe750e"} Oct 30 02:15:20.536707 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:20.536610 6221 scope.go:122] "RemoveContainer" containerID="1c1bcc4b8e6c97310cf7d0ee2751730ba04f81f1354d9e805aa1721e0ef25949" Oct 30 02:15:20.538509 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:20.538483 6221 generic.go:356] "Generic (PLEG): container finished" podID="ebe12490-67c0-4d1d-ad75-1ade885d5650" containerID="09d17e83cfccba3109c2babe5683f3284cef7471a3334bcbe50b4600f27c9391" exitCode=60 Oct 30 02:15:20.538672 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:20.538553 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-601d4b-29888775-d8mmg" event={"ID":"ebe12490-67c0-4d1d-ad75-1ade885d5650","Type":"ContainerDied","Data":"09d17e83cfccba3109c2babe5683f3284cef7471a3334bcbe50b4600f27c9391"} Oct 30 02:15:20.538898 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:20.538883 6221 scope.go:122] "RemoveContainer" containerID="09d17e83cfccba3109c2babe5683f3284cef7471a3334bcbe50b4600f27c9391" Oct 30 02:15:20.539174 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:15:20.539155 6221 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-601d4b-29888775-d8mmg_odh-ai-gateway-infra(ebe12490-67c0-4d1d-ad75-1ade885d5650)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-601d4b-29888775-d8mmg" podUID="ebe12490-67c0-4d1d-ad75-1ade885d5650" Oct 30 02:15:20.541992 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:20.541967 6221 generic.go:356] "Generic (PLEG): container finished" podID="144c527d-0189-4c40-a64a-a207dfbdd710" containerID="3b863fa5e4ad760c3d169f15e5690690434ad29752da105c53373175f5819e40" exitCode=60 Oct 30 02:15:20.545631 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:20.545612 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888775-jrp5d" Oct 30 02:15:20.547177 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:20.547150 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22e99d-29888775-kbvl4" Oct 30 02:15:20.548792 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:20.548773 6221 scope.go:122] "RemoveContainer" containerID="0ba233234c5960a613f5f35d205dcbf17f479850d9c7de263ad26479f7682336" Oct 30 02:15:20.552310 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:20.552289 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68d8d09d-2725-417b-8d9d-c0ea49ce29ac" path="/var/lib/kubelet/pods/68d8d09d-2725-417b-8d9d-c0ea49ce29ac/volumes" Oct 30 02:15:20.552767 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:20.552747 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a506dae9-1ced-409d-99db-413b66dd9d59" path="/var/lib/kubelet/pods/a506dae9-1ced-409d-99db-413b66dd9d59/volumes" Oct 30 02:15:20.553173 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:20.553160 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4ac5f29-bf87-42b2-82af-ee49cd16e879" path="/var/lib/kubelet/pods/c4ac5f29-bf87-42b2-82af-ee49cd16e879/volumes" Oct 30 02:15:20.554225 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:20.554201 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888775-mwqtg" event={"ID":"144c527d-0189-4c40-a64a-a207dfbdd710","Type":"ContainerDied","Data":"3b863fa5e4ad760c3d169f15e5690690434ad29752da105c53373175f5819e40"} Oct 30 02:15:20.554352 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:20.554252 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888775-jrp5d" event={"ID":"66320d4a-0c8f-483d-98e4-b17658b6b27a","Type":"ContainerDied","Data":"93da2a896cb5e6f491bfd2e64cf3e624d8cec3eb32fe5785a4da09a3777a94ab"} Oct 30 02:15:20.554352 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:20.554280 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22e99d-29888775-kbvl4" event={"ID":"d90c1ed4-5a28-4ce1-ba6e-c521698c3667","Type":"ContainerDied","Data":"ebec57363ced8d43445e91b2e7671b7b5c87a983613afb0dd7099687ce4a16f6"} Oct 30 02:15:20.554671 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:20.554653 6221 scope.go:122] "RemoveContainer" containerID="3b863fa5e4ad760c3d169f15e5690690434ad29752da105c53373175f5819e40" Oct 30 02:15:20.554943 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:15:20.554921 6221 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-76a19e-29888775-mwqtg_odh-ai-gateway-infra(144c527d-0189-4c40-a64a-a207dfbdd710)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888775-mwqtg" podUID="144c527d-0189-4c40-a64a-a207dfbdd710" Oct 30 02:15:20.566218 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:20.566197 6221 scope.go:122] "RemoveContainer" containerID="4af24c08dd7b197e07edcacb8f8771b481a2b0ef7ff840bd84949a5a1cdaf496" Oct 30 02:15:20.566936 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:20.566893 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-601d4b-29888775-d8mmg"] Oct 30 02:15:20.580523 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:20.580506 6221 scope.go:122] "RemoveContainer" containerID="ec49438ab901962861e75a9f970a4adef0bce5a1d54fb5e216aa940642dff642" Oct 30 02:15:20.590785 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:20.590762 6221 scope.go:122] "RemoveContainer" containerID="ec8cce93a44657c3413340b86621165538e9272585af5dff22b6677d95e3e58e" Oct 30 02:15:20.610294 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:20.610273 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888775-mwqtg"] Oct 30 02:15:20.629742 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:20.629719 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22e99d-29888775-kbvl4"] Oct 30 02:15:20.639238 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:20.639214 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-22e99d-29888775-kbvl4"] Oct 30 02:15:20.757628 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:20.757596 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888775-x7l7p"] Oct 30 02:15:20.760720 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:20.760698 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-eda881-29888775-x7l7p"] Oct 30 02:15:20.795515 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:20.795429 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888775-jrp5d"] Oct 30 02:15:20.798934 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:20.798914 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2e618f-29888775-jrp5d"] Oct 30 02:15:21.423583 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:21.423171 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cd8173-5674dfcc9f-sqswj" podUID="a08e7d0c-aaf7-4960-bc25-23e2290c1bd6" containerName="payload-pre-processing" containerID="cri-o://b9fbebb8f49d5e6c57a95b7244372e0a697cd501df678fe2ff23771462cdb5ef" gracePeriod=25 Oct 30 02:15:21.504748 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:15:21.504714 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 02:15:21.570722 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:21.570690 6221 generic.go:356] "Generic (PLEG): container finished" podID="def2849e-c94c-4685-9992-e1121b744ac8" containerID="355c5b91826c434e36396766ecd25141bbb931f9a1ae495f9b0ff6ddc5603c45" exitCode=60 Oct 30 02:15:21.570982 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:21.570784 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888775-8zqzh" event={"ID":"def2849e-c94c-4685-9992-e1121b744ac8","Type":"ContainerDied","Data":"355c5b91826c434e36396766ecd25141bbb931f9a1ae495f9b0ff6ddc5603c45"} Oct 30 02:15:21.570982 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:21.570842 6221 scope.go:122] "RemoveContainer" containerID="e8c4bf6e85a819619feb19d49dd7c21de4e1835b148b23268e10ae9d0963a493" Oct 30 02:15:21.571349 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:21.571330 6221 scope.go:122] "RemoveContainer" containerID="355c5b91826c434e36396766ecd25141bbb931f9a1ae495f9b0ff6ddc5603c45" Oct 30 02:15:21.571867 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:15:21.571712 6221 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888775-8zqzh_odh-ai-gateway-infra(def2849e-c94c-4685-9992-e1121b744ac8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888775-8zqzh" podUID="def2849e-c94c-4685-9992-e1121b744ac8" Oct 30 02:15:21.607016 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:21.606981 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888775-8zqzh"] Oct 30 02:15:21.963628 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:21.963598 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-601d4b-29888775-d8mmg" Oct 30 02:15:21.999524 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:21.999489 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888775-mwqtg" Oct 30 02:15:22.043201 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:22.043159 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/144c527d-0189-4c40-a64a-a207dfbdd710-kube-api-access-xgqvk\" (UniqueName: \"kubernetes.io/projected/144c527d-0189-4c40-a64a-a207dfbdd710-kube-api-access-xgqvk\") pod \"144c527d-0189-4c40-a64a-a207dfbdd710\" (UID: \"144c527d-0189-4c40-a64a-a207dfbdd710\") " Oct 30 02:15:22.043411 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:22.043234 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebe12490-67c0-4d1d-ad75-1ade885d5650-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe12490-67c0-4d1d-ad75-1ade885d5650-maas-api-service-ca\") pod \"ebe12490-67c0-4d1d-ad75-1ade885d5650\" (UID: \"ebe12490-67c0-4d1d-ad75-1ade885d5650\") " Oct 30 02:15:22.043517 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:22.043476 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebe12490-67c0-4d1d-ad75-1ade885d5650-kube-api-access-ff9b7\" (UniqueName: \"kubernetes.io/projected/ebe12490-67c0-4d1d-ad75-1ade885d5650-kube-api-access-ff9b7\") pod \"ebe12490-67c0-4d1d-ad75-1ade885d5650\" (UID: \"ebe12490-67c0-4d1d-ad75-1ade885d5650\") " Oct 30 02:15:22.043517 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:22.043503 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/144c527d-0189-4c40-a64a-a207dfbdd710-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/144c527d-0189-4c40-a64a-a207dfbdd710-maas-api-service-ca\") pod \"144c527d-0189-4c40-a64a-a207dfbdd710\" (UID: \"144c527d-0189-4c40-a64a-a207dfbdd710\") " Oct 30 02:15:22.043674 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:22.043646 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebe12490-67c0-4d1d-ad75-1ade885d5650-maas-api-service-ca" pod "ebe12490-67c0-4d1d-ad75-1ade885d5650" (UID: "ebe12490-67c0-4d1d-ad75-1ade885d5650"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:22.043968 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:22.043946 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/144c527d-0189-4c40-a64a-a207dfbdd710-maas-api-service-ca" pod "144c527d-0189-4c40-a64a-a207dfbdd710" (UID: "144c527d-0189-4c40-a64a-a207dfbdd710"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:22.044157 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:22.044139 6221 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebe12490-67c0-4d1d-ad75-1ade885d5650-maas-api-service-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:22.044249 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:22.044162 6221 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/144c527d-0189-4c40-a64a-a207dfbdd710-maas-api-service-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:22.045567 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:22.045524 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/144c527d-0189-4c40-a64a-a207dfbdd710-kube-api-access-xgqvk" pod "144c527d-0189-4c40-a64a-a207dfbdd710" (UID: "144c527d-0189-4c40-a64a-a207dfbdd710"). InnerVolumeSpecName "kube-api-access-xgqvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:22.045811 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:22.045636 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebe12490-67c0-4d1d-ad75-1ade885d5650-kube-api-access-ff9b7" pod "ebe12490-67c0-4d1d-ad75-1ade885d5650" (UID: "ebe12490-67c0-4d1d-ad75-1ade885d5650"). InnerVolumeSpecName "kube-api-access-ff9b7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:22.145232 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:22.145194 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff9b7\" (UniqueName: \"kubernetes.io/projected/ebe12490-67c0-4d1d-ad75-1ade885d5650-kube-api-access-ff9b7\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:22.145232 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:22.145225 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgqvk\" (UniqueName: \"kubernetes.io/projected/144c527d-0189-4c40-a64a-a207dfbdd710-kube-api-access-xgqvk\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:22.549695 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:22.549661 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66320d4a-0c8f-483d-98e4-b17658b6b27a" path="/var/lib/kubelet/pods/66320d4a-0c8f-483d-98e4-b17658b6b27a/volumes" Oct 30 02:15:22.550176 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:22.550162 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9" path="/var/lib/kubelet/pods/cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9/volumes" Oct 30 02:15:22.550588 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:22.550575 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d90c1ed4-5a28-4ce1-ba6e-c521698c3667" path="/var/lib/kubelet/pods/d90c1ed4-5a28-4ce1-ba6e-c521698c3667/volumes" Oct 30 02:15:22.599285 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:22.599263 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-601d4b-29888775-d8mmg" Oct 30 02:15:22.599434 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:22.599263 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-601d4b-29888775-d8mmg" event={"ID":"ebe12490-67c0-4d1d-ad75-1ade885d5650","Type":"ContainerDied","Data":"3536aa7411e3ed92a3c7c9daac5627b596f5e03e954b6b1609837575c3b62b23"} Oct 30 02:15:22.599434 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:22.599403 6221 scope.go:122] "RemoveContainer" containerID="09d17e83cfccba3109c2babe5683f3284cef7471a3334bcbe50b4600f27c9391" Oct 30 02:15:22.600823 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:22.600803 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888775-mwqtg" Oct 30 02:15:22.600957 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:22.600802 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888775-mwqtg" event={"ID":"144c527d-0189-4c40-a64a-a207dfbdd710","Type":"ContainerDied","Data":"118c846dc6663fecafd3511c728520893787b59c2954ad67df3ddbc0de8915f0"} Oct 30 02:15:22.611798 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:22.611778 6221 scope.go:122] "RemoveContainer" containerID="3b863fa5e4ad760c3d169f15e5690690434ad29752da105c53373175f5819e40" Oct 30 02:15:22.680118 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:22.680083 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888775-mwqtg"] Oct 30 02:15:22.688991 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:22.688958 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-76a19e-29888775-mwqtg"] Oct 30 02:15:22.727510 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:22.727476 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-601d4b-29888775-d8mmg"] Oct 30 02:15:22.738319 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:22.738291 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-601d4b-29888775-d8mmg"] Oct 30 02:15:22.759014 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:22.758982 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888775-8zqzh" Oct 30 02:15:22.854673 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:22.854587 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/def2849e-c94c-4685-9992-e1121b744ac8-kube-api-access-8h9fj\" (UniqueName: \"kubernetes.io/projected/def2849e-c94c-4685-9992-e1121b744ac8-kube-api-access-8h9fj\") pod \"def2849e-c94c-4685-9992-e1121b744ac8\" (UID: \"def2849e-c94c-4685-9992-e1121b744ac8\") " Oct 30 02:15:22.854856 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:22.854692 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/def2849e-c94c-4685-9992-e1121b744ac8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/def2849e-c94c-4685-9992-e1121b744ac8-maas-api-service-ca\") pod \"def2849e-c94c-4685-9992-e1121b744ac8\" (UID: \"def2849e-c94c-4685-9992-e1121b744ac8\") " Oct 30 02:15:22.855097 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:22.855074 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/def2849e-c94c-4685-9992-e1121b744ac8-maas-api-service-ca" pod "def2849e-c94c-4685-9992-e1121b744ac8" (UID: "def2849e-c94c-4685-9992-e1121b744ac8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:22.855603 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:22.855580 6221 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/def2849e-c94c-4685-9992-e1121b744ac8-maas-api-service-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:22.857224 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:22.857197 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/def2849e-c94c-4685-9992-e1121b744ac8-kube-api-access-8h9fj" pod "def2849e-c94c-4685-9992-e1121b744ac8" (UID: "def2849e-c94c-4685-9992-e1121b744ac8"). InnerVolumeSpecName "kube-api-access-8h9fj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:22.956476 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:22.956443 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h9fj\" (UniqueName: \"kubernetes.io/projected/def2849e-c94c-4685-9992-e1121b744ac8-kube-api-access-8h9fj\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:23.610049 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:23.610020 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888775-8zqzh" Oct 30 02:15:23.610564 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:23.610024 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888775-8zqzh" event={"ID":"def2849e-c94c-4685-9992-e1121b744ac8","Type":"ContainerDied","Data":"18ccb5511a535f22e4bf02f93123c92687cb202fc1bc4930a4d3c56d24291c7f"} Oct 30 02:15:23.610564 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:23.610172 6221 scope.go:122] "RemoveContainer" containerID="355c5b91826c434e36396766ecd25141bbb931f9a1ae495f9b0ff6ddc5603c45" Oct 30 02:15:23.667650 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:23.667619 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888775-8zqzh"] Oct 30 02:15:23.675671 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:23.675647 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-4f7628-29888775-8zqzh"] Oct 30 02:15:24.549824 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:24.549793 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="144c527d-0189-4c40-a64a-a207dfbdd710" path="/var/lib/kubelet/pods/144c527d-0189-4c40-a64a-a207dfbdd710/volumes" Oct 30 02:15:24.550235 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:24.550221 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="def2849e-c94c-4685-9992-e1121b744ac8" path="/var/lib/kubelet/pods/def2849e-c94c-4685-9992-e1121b744ac8/volumes" Oct 30 02:15:24.550640 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:24.550626 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebe12490-67c0-4d1d-ad75-1ade885d5650" path="/var/lib/kubelet/pods/ebe12490-67c0-4d1d-ad75-1ade885d5650/volumes" Oct 30 02:15:25.433038 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:25.433001 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cd8173-5674dfcc9f-sqswj" podUID="a08e7d0c-aaf7-4960-bc25-23e2290c1bd6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:15:29.055325 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:29.055288 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2"] Oct 30 02:15:29.064059 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:29.055610 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" podUID="8e229797-d116-4670-86e9-e446396807ad" containerName="istio-proxy" containerID="cri-o://61510b829862039589f3a3fcd126a95f6d4b12c782c30bc68dcb8748c3224fae" gracePeriod=30 Oct 30 02:15:34.330190 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:34.330123 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" Oct 30 02:15:34.464509 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:34.464473 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e229797-d116-4670-86e9-e446396807ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e229797-d116-4670-86e9-e446396807ad-istiod-ca-cert\") pod \"8e229797-d116-4670-86e9-e446396807ad\" (UID: \"8e229797-d116-4670-86e9-e446396807ad\") " Oct 30 02:15:34.464742 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:34.464522 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e229797-d116-4670-86e9-e446396807ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e229797-d116-4670-86e9-e446396807ad-istio-envoy\") pod \"8e229797-d116-4670-86e9-e446396807ad\" (UID: \"8e229797-d116-4670-86e9-e446396807ad\") " Oct 30 02:15:34.464742 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:34.464593 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e229797-d116-4670-86e9-e446396807ad-kube-api-access-sblpl\" (UniqueName: \"kubernetes.io/projected/8e229797-d116-4670-86e9-e446396807ad-kube-api-access-sblpl\") pod \"8e229797-d116-4670-86e9-e446396807ad\" (UID: \"8e229797-d116-4670-86e9-e446396807ad\") " Oct 30 02:15:34.464742 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:34.464643 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e229797-d116-4670-86e9-e446396807ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e229797-d116-4670-86e9-e446396807ad-workload-certs\") pod \"8e229797-d116-4670-86e9-e446396807ad\" (UID: \"8e229797-d116-4670-86e9-e446396807ad\") " Oct 30 02:15:34.464742 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:34.464717 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e229797-d116-4670-86e9-e446396807ad-istio-token\" (UniqueName: \"kubernetes.io/projected/8e229797-d116-4670-86e9-e446396807ad-istio-token\") pod \"8e229797-d116-4670-86e9-e446396807ad\" (UID: \"8e229797-d116-4670-86e9-e446396807ad\") " Oct 30 02:15:34.464995 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:34.464903 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e229797-d116-4670-86e9-e446396807ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e229797-d116-4670-86e9-e446396807ad-credential-socket\") pod \"8e229797-d116-4670-86e9-e446396807ad\" (UID: \"8e229797-d116-4670-86e9-e446396807ad\") " Oct 30 02:15:34.464995 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:34.464927 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e229797-d116-4670-86e9-e446396807ad-workload-certs" pod "8e229797-d116-4670-86e9-e446396807ad" (UID: "8e229797-d116-4670-86e9-e446396807ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:15:34.465107 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:34.465044 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e229797-d116-4670-86e9-e446396807ad-istiod-ca-cert" pod "8e229797-d116-4670-86e9-e446396807ad" (UID: "8e229797-d116-4670-86e9-e446396807ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:34.465171 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:34.465154 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e229797-d116-4670-86e9-e446396807ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e229797-d116-4670-86e9-e446396807ad-workload-socket\") pod \"8e229797-d116-4670-86e9-e446396807ad\" (UID: \"8e229797-d116-4670-86e9-e446396807ad\") " Oct 30 02:15:34.465223 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:34.465164 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e229797-d116-4670-86e9-e446396807ad-credential-socket" pod "8e229797-d116-4670-86e9-e446396807ad" (UID: "8e229797-d116-4670-86e9-e446396807ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:15:34.465223 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:34.465188 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e229797-d116-4670-86e9-e446396807ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e229797-d116-4670-86e9-e446396807ad-istio-data\") pod \"8e229797-d116-4670-86e9-e446396807ad\" (UID: \"8e229797-d116-4670-86e9-e446396807ad\") " Oct 30 02:15:34.465307 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:34.465257 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e229797-d116-4670-86e9-e446396807ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e229797-d116-4670-86e9-e446396807ad-istio-podinfo\") pod \"8e229797-d116-4670-86e9-e446396807ad\" (UID: \"8e229797-d116-4670-86e9-e446396807ad\") " Oct 30 02:15:34.465365 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:34.465353 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e229797-d116-4670-86e9-e446396807ad-workload-socket" pod "8e229797-d116-4670-86e9-e446396807ad" (UID: "8e229797-d116-4670-86e9-e446396807ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:15:34.465614 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:34.465581 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e229797-d116-4670-86e9-e446396807ad-istio-data" pod "8e229797-d116-4670-86e9-e446396807ad" (UID: "8e229797-d116-4670-86e9-e446396807ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:15:34.465967 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:34.465945 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e229797-d116-4670-86e9-e446396807ad-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:34.466054 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:34.465972 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e229797-d116-4670-86e9-e446396807ad-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:34.466054 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:34.465987 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e229797-d116-4670-86e9-e446396807ad-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:34.466054 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:34.466002 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e229797-d116-4670-86e9-e446396807ad-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:34.466054 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:34.466015 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e229797-d116-4670-86e9-e446396807ad-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:34.467058 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:34.467036 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e229797-d116-4670-86e9-e446396807ad-istio-envoy" pod "8e229797-d116-4670-86e9-e446396807ad" (UID: "8e229797-d116-4670-86e9-e446396807ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:15:34.467454 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:34.467431 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e229797-d116-4670-86e9-e446396807ad-istio-podinfo" pod "8e229797-d116-4670-86e9-e446396807ad" (UID: "8e229797-d116-4670-86e9-e446396807ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:15:34.467548 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:34.467447 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e229797-d116-4670-86e9-e446396807ad-istio-token" pod "8e229797-d116-4670-86e9-e446396807ad" (UID: "8e229797-d116-4670-86e9-e446396807ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:34.467548 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:34.467485 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e229797-d116-4670-86e9-e446396807ad-kube-api-access-sblpl" pod "8e229797-d116-4670-86e9-e446396807ad" (UID: "8e229797-d116-4670-86e9-e446396807ad"). InnerVolumeSpecName "kube-api-access-sblpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:34.567137 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:34.567106 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e229797-d116-4670-86e9-e446396807ad-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:34.567137 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:34.567137 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e229797-d116-4670-86e9-e446396807ad-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:34.567307 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:34.567151 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sblpl\" (UniqueName: \"kubernetes.io/projected/8e229797-d116-4670-86e9-e446396807ad-kube-api-access-sblpl\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:34.567307 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:34.567164 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e229797-d116-4670-86e9-e446396807ad-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:34.725878 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:34.725788 6221 generic.go:356] "Generic (PLEG): container finished" podID="8e229797-d116-4670-86e9-e446396807ad" containerID="61510b829862039589f3a3fcd126a95f6d4b12c782c30bc68dcb8748c3224fae" exitCode=0 Oct 30 02:15:34.725878 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:34.725845 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" Oct 30 02:15:34.726074 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:34.725873 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" event={"ID":"8e229797-d116-4670-86e9-e446396807ad","Type":"ContainerDied","Data":"61510b829862039589f3a3fcd126a95f6d4b12c782c30bc68dcb8748c3224fae"} Oct 30 02:15:34.726074 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:34.725933 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2" event={"ID":"8e229797-d116-4670-86e9-e446396807ad","Type":"ContainerDied","Data":"ee05c22f08e59944356c7eef459b1c642ee629f65059260fb369f8262a93eace"} Oct 30 02:15:34.726074 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:34.725966 6221 scope.go:122] "RemoveContainer" containerID="61510b829862039589f3a3fcd126a95f6d4b12c782c30bc68dcb8748c3224fae" Oct 30 02:15:34.740424 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:34.740377 6221 scope.go:122] "RemoveContainer" containerID="4a160237c9824cc92ad28b8c6d3c3961a6c0292b598bbf64354fc22f4219746b" Oct 30 02:15:34.750723 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:34.750701 6221 scope.go:122] "RemoveContainer" containerID="61510b829862039589f3a3fcd126a95f6d4b12c782c30bc68dcb8748c3224fae" Oct 30 02:15:34.750999 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:15:34.750979 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61510b829862039589f3a3fcd126a95f6d4b12c782c30bc68dcb8748c3224fae\": container with ID starting with 61510b829862039589f3a3fcd126a95f6d4b12c782c30bc68dcb8748c3224fae not found: ID does not exist" containerID="61510b829862039589f3a3fcd126a95f6d4b12c782c30bc68dcb8748c3224fae" Oct 30 02:15:34.751070 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:34.751008 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61510b829862039589f3a3fcd126a95f6d4b12c782c30bc68dcb8748c3224fae"} err="failed to get container status \"61510b829862039589f3a3fcd126a95f6d4b12c782c30bc68dcb8748c3224fae\": rpc error: code = NotFound desc = could not find container \"61510b829862039589f3a3fcd126a95f6d4b12c782c30bc68dcb8748c3224fae\": container with ID starting with 61510b829862039589f3a3fcd126a95f6d4b12c782c30bc68dcb8748c3224fae not found: ID does not exist" Oct 30 02:15:34.751070 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:34.751027 6221 scope.go:122] "RemoveContainer" containerID="4a160237c9824cc92ad28b8c6d3c3961a6c0292b598bbf64354fc22f4219746b" Oct 30 02:15:34.751255 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:15:34.751238 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a160237c9824cc92ad28b8c6d3c3961a6c0292b598bbf64354fc22f4219746b\": container with ID starting with 4a160237c9824cc92ad28b8c6d3c3961a6c0292b598bbf64354fc22f4219746b not found: ID does not exist" containerID="4a160237c9824cc92ad28b8c6d3c3961a6c0292b598bbf64354fc22f4219746b" Oct 30 02:15:34.751297 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:34.751259 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a160237c9824cc92ad28b8c6d3c3961a6c0292b598bbf64354fc22f4219746b"} err="failed to get container status \"4a160237c9824cc92ad28b8c6d3c3961a6c0292b598bbf64354fc22f4219746b\": rpc error: code = NotFound desc = could not find container \"4a160237c9824cc92ad28b8c6d3c3961a6c0292b598bbf64354fc22f4219746b\": container with ID starting with 4a160237c9824cc92ad28b8c6d3c3961a6c0292b598bbf64354fc22f4219746b not found: ID does not exist" Oct 30 02:15:34.780644 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:34.780620 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2"] Oct 30 02:15:34.792660 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:34.792633 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-292018-openshift-default-874d9cb87-lk4k2"] Oct 30 02:15:35.432989 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:35.432955 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cd8173-5674dfcc9f-sqswj" podUID="a08e7d0c-aaf7-4960-bc25-23e2290c1bd6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:15:36.550885 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:36.550852 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e229797-d116-4670-86e9-e446396807ad" path="/var/lib/kubelet/pods/8e229797-d116-4670-86e9-e446396807ad/volumes" Oct 30 02:15:41.727444 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:41.727410 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cd8173-5674dfcc9f-sqswj" Oct 30 02:15:41.819123 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:41.819089 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a08e7d0c-aaf7-4960-bc25-23e2290c1bd6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a08e7d0c-aaf7-4960-bc25-23e2290c1bd6-plugins-config-volume\") pod \"a08e7d0c-aaf7-4960-bc25-23e2290c1bd6\" (UID: \"a08e7d0c-aaf7-4960-bc25-23e2290c1bd6\") " Oct 30 02:15:41.819397 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:41.819376 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a08e7d0c-aaf7-4960-bc25-23e2290c1bd6-kube-api-access-jnl8c\" (UniqueName: \"kubernetes.io/projected/a08e7d0c-aaf7-4960-bc25-23e2290c1bd6-kube-api-access-jnl8c\") pod \"a08e7d0c-aaf7-4960-bc25-23e2290c1bd6\" (UID: \"a08e7d0c-aaf7-4960-bc25-23e2290c1bd6\") " Oct 30 02:15:41.820128 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:41.820102 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a08e7d0c-aaf7-4960-bc25-23e2290c1bd6-plugins-config-volume" pod "a08e7d0c-aaf7-4960-bc25-23e2290c1bd6" (UID: "a08e7d0c-aaf7-4960-bc25-23e2290c1bd6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:41.820430 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:41.820402 6221 generic.go:356] "Generic (PLEG): container finished" podID="a08e7d0c-aaf7-4960-bc25-23e2290c1bd6" containerID="b9fbebb8f49d5e6c57a95b7244372e0a697cd501df678fe2ff23771462cdb5ef" exitCode=0 Oct 30 02:15:41.820570 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:41.820418 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cd8173-5674dfcc9f-sqswj" Oct 30 02:15:41.820652 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:41.820587 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cd8173-5674dfcc9f-sqswj" event={"ID":"a08e7d0c-aaf7-4960-bc25-23e2290c1bd6","Type":"ContainerDied","Data":"b9fbebb8f49d5e6c57a95b7244372e0a697cd501df678fe2ff23771462cdb5ef"} Oct 30 02:15:41.820719 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:41.820676 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-cd8173-5674dfcc9f-sqswj" event={"ID":"a08e7d0c-aaf7-4960-bc25-23e2290c1bd6","Type":"ContainerDied","Data":"961688c58d3ecd1d3fd3e84602010533bfd5dee24e589c394c7708b5db444e5b"} Oct 30 02:15:41.820719 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:41.820714 6221 scope.go:122] "RemoveContainer" containerID="b9fbebb8f49d5e6c57a95b7244372e0a697cd501df678fe2ff23771462cdb5ef" Oct 30 02:15:41.823440 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:41.823417 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a08e7d0c-aaf7-4960-bc25-23e2290c1bd6-kube-api-access-jnl8c" pod "a08e7d0c-aaf7-4960-bc25-23e2290c1bd6" (UID: "a08e7d0c-aaf7-4960-bc25-23e2290c1bd6"). InnerVolumeSpecName "kube-api-access-jnl8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:41.843951 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:41.843791 6221 scope.go:122] "RemoveContainer" containerID="b9fbebb8f49d5e6c57a95b7244372e0a697cd501df678fe2ff23771462cdb5ef" Oct 30 02:15:41.844206 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:15:41.844182 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9fbebb8f49d5e6c57a95b7244372e0a697cd501df678fe2ff23771462cdb5ef\": container with ID starting with b9fbebb8f49d5e6c57a95b7244372e0a697cd501df678fe2ff23771462cdb5ef not found: ID does not exist" containerID="b9fbebb8f49d5e6c57a95b7244372e0a697cd501df678fe2ff23771462cdb5ef" Oct 30 02:15:41.844335 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:41.844213 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9fbebb8f49d5e6c57a95b7244372e0a697cd501df678fe2ff23771462cdb5ef"} err="failed to get container status \"b9fbebb8f49d5e6c57a95b7244372e0a697cd501df678fe2ff23771462cdb5ef\": rpc error: code = NotFound desc = could not find container \"b9fbebb8f49d5e6c57a95b7244372e0a697cd501df678fe2ff23771462cdb5ef\": container with ID starting with b9fbebb8f49d5e6c57a95b7244372e0a697cd501df678fe2ff23771462cdb5ef not found: ID does not exist" Oct 30 02:15:41.921452 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:41.921421 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnl8c\" (UniqueName: \"kubernetes.io/projected/a08e7d0c-aaf7-4960-bc25-23e2290c1bd6-kube-api-access-jnl8c\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:41.921452 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:41.921452 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a08e7d0c-aaf7-4960-bc25-23e2290c1bd6-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:42.167618 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:42.167584 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-cd8173-5674dfcc9f-sqswj"] Oct 30 02:15:42.170737 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:42.170710 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-cd8173-5674dfcc9f-sqswj"] Oct 30 02:15:42.548769 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:42.548738 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a08e7d0c-aaf7-4960-bc25-23e2290c1bd6" path="/var/lib/kubelet/pods/a08e7d0c-aaf7-4960-bc25-23e2290c1bd6/volumes" Oct 30 02:15:51.494521 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:15:51.494473 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 02:15:52.296577 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:52.296541 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd8173-7b89fb6b85-rw4zt"] Oct 30 02:15:52.296835 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:52.296807 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd8173-7b89fb6b85-rw4zt" podUID="0eec39a3-c8c5-47fd-b1c3-132da72e7b65" containerName="maas-api" containerID="cri-o://59e292a5bc4003538ee23a517aa00cd0dab55115b8d8abc739b40e7d9314f84e" gracePeriod=30 Oct 30 02:15:52.742155 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:52.741800 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd8173-7b89fb6b85-rw4zt" Oct 30 02:15:52.805181 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:52.805148 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0eec39a3-c8c5-47fd-b1c3-132da72e7b65-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0eec39a3-c8c5-47fd-b1c3-132da72e7b65-metrics-certs\") pod \"0eec39a3-c8c5-47fd-b1c3-132da72e7b65\" (UID: \"0eec39a3-c8c5-47fd-b1c3-132da72e7b65\") " Oct 30 02:15:52.805617 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:52.805585 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0eec39a3-c8c5-47fd-b1c3-132da72e7b65-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0eec39a3-c8c5-47fd-b1c3-132da72e7b65-maas-api-tls\") pod \"0eec39a3-c8c5-47fd-b1c3-132da72e7b65\" (UID: \"0eec39a3-c8c5-47fd-b1c3-132da72e7b65\") " Oct 30 02:15:52.806905 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:52.806884 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0eec39a3-c8c5-47fd-b1c3-132da72e7b65-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eec39a3-c8c5-47fd-b1c3-132da72e7b65-openshift-service-ca\") pod \"0eec39a3-c8c5-47fd-b1c3-132da72e7b65\" (UID: \"0eec39a3-c8c5-47fd-b1c3-132da72e7b65\") " Oct 30 02:15:52.807500 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:52.807370 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eec39a3-c8c5-47fd-b1c3-132da72e7b65-openshift-service-ca" pod "0eec39a3-c8c5-47fd-b1c3-132da72e7b65" (UID: "0eec39a3-c8c5-47fd-b1c3-132da72e7b65"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:15:52.807933 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:52.807914 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eec39a3-c8c5-47fd-b1c3-132da72e7b65-kube-api-access-b899z\" (UniqueName: \"kubernetes.io/projected/0eec39a3-c8c5-47fd-b1c3-132da72e7b65-kube-api-access-b899z\") pod \"0eec39a3-c8c5-47fd-b1c3-132da72e7b65\" (UID: \"0eec39a3-c8c5-47fd-b1c3-132da72e7b65\") " Oct 30 02:15:52.810428 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:52.810410 6221 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0eec39a3-c8c5-47fd-b1c3-132da72e7b65-openshift-service-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:52.819146 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:52.819099 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eec39a3-c8c5-47fd-b1c3-132da72e7b65-metrics-certs" pod "0eec39a3-c8c5-47fd-b1c3-132da72e7b65" (UID: "0eec39a3-c8c5-47fd-b1c3-132da72e7b65"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:15:52.821846 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:52.821780 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eec39a3-c8c5-47fd-b1c3-132da72e7b65-kube-api-access-b899z" pod "0eec39a3-c8c5-47fd-b1c3-132da72e7b65" (UID: "0eec39a3-c8c5-47fd-b1c3-132da72e7b65"). InnerVolumeSpecName "kube-api-access-b899z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:15:52.822935 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:52.822896 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eec39a3-c8c5-47fd-b1c3-132da72e7b65-maas-api-tls" pod "0eec39a3-c8c5-47fd-b1c3-132da72e7b65" (UID: "0eec39a3-c8c5-47fd-b1c3-132da72e7b65"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:15:52.914252 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:52.913975 6221 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0eec39a3-c8c5-47fd-b1c3-132da72e7b65-metrics-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:52.914252 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:52.914010 6221 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0eec39a3-c8c5-47fd-b1c3-132da72e7b65-maas-api-tls\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:52.914252 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:52.914025 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b899z\" (UniqueName: \"kubernetes.io/projected/0eec39a3-c8c5-47fd-b1c3-132da72e7b65-kube-api-access-b899z\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:15:52.936017 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:52.935979 6221 generic.go:356] "Generic (PLEG): container finished" podID="0eec39a3-c8c5-47fd-b1c3-132da72e7b65" containerID="59e292a5bc4003538ee23a517aa00cd0dab55115b8d8abc739b40e7d9314f84e" exitCode=0 Oct 30 02:15:52.936186 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:52.936035 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd8173-7b89fb6b85-rw4zt" Oct 30 02:15:52.936186 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:52.936066 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd8173-7b89fb6b85-rw4zt" event={"ID":"0eec39a3-c8c5-47fd-b1c3-132da72e7b65","Type":"ContainerDied","Data":"59e292a5bc4003538ee23a517aa00cd0dab55115b8d8abc739b40e7d9314f84e"} Oct 30 02:15:52.936186 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:52.936128 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd8173-7b89fb6b85-rw4zt" event={"ID":"0eec39a3-c8c5-47fd-b1c3-132da72e7b65","Type":"ContainerDied","Data":"3921a026fb0dde637d46e72a172aeda3ae3056d71b552edce4da61f4586a6f93"} Oct 30 02:15:52.936186 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:52.936154 6221 scope.go:122] "RemoveContainer" containerID="59e292a5bc4003538ee23a517aa00cd0dab55115b8d8abc739b40e7d9314f84e" Oct 30 02:15:52.947938 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:52.947919 6221 scope.go:122] "RemoveContainer" containerID="59e292a5bc4003538ee23a517aa00cd0dab55115b8d8abc739b40e7d9314f84e" Oct 30 02:15:52.948301 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:15:52.948275 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59e292a5bc4003538ee23a517aa00cd0dab55115b8d8abc739b40e7d9314f84e\": container with ID starting with 59e292a5bc4003538ee23a517aa00cd0dab55115b8d8abc739b40e7d9314f84e not found: ID does not exist" containerID="59e292a5bc4003538ee23a517aa00cd0dab55115b8d8abc739b40e7d9314f84e" Oct 30 02:15:52.948389 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:52.948311 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59e292a5bc4003538ee23a517aa00cd0dab55115b8d8abc739b40e7d9314f84e"} err="failed to get container status \"59e292a5bc4003538ee23a517aa00cd0dab55115b8d8abc739b40e7d9314f84e\": rpc error: code = NotFound desc = could not find container \"59e292a5bc4003538ee23a517aa00cd0dab55115b8d8abc739b40e7d9314f84e\": container with ID starting with 59e292a5bc4003538ee23a517aa00cd0dab55115b8d8abc739b40e7d9314f84e not found: ID does not exist" Oct 30 02:15:52.988720 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:52.988696 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd8173-7b89fb6b85-rw4zt"] Oct 30 02:15:52.994474 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:52.994449 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-cd8173-7b89fb6b85-rw4zt"] Oct 30 02:15:54.548961 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:15:54.548929 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eec39a3-c8c5-47fd-b1c3-132da72e7b65" path="/var/lib/kubelet/pods/0eec39a3-c8c5-47fd-b1c3-132da72e7b65/volumes" Oct 30 02:16:01.908913 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:01.908875 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 02:16:10.694701 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:10.694668 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b6b7fc-68dff5d4d-295gs"] Oct 30 02:16:10.725161 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:10.725129 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b6b7fc-546d4d784c-6cjjd"] Oct 30 02:16:15.695297 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:15.695251 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b6b7fc-68dff5d4d-295gs" podUID="a36fc050-879a-4182-ac44-6d3ad2ff189e" containerName="payload-processing" containerID="cri-o://8a81ae121384c2eae6168aa3a7649f9dfc706763a4a454c9c2e956a5d966cd44" gracePeriod=25 Oct 30 02:16:15.726048 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:15.726005 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b6b7fc-546d4d784c-6cjjd" podUID="9061008c-04d0-42d9-96ae-a8ebe16c54b7" containerName="payload-pre-processing" containerID="cri-o://76444066abf01df9d4275cd0ef6c2264589cb0712c1360fe7b6d9aff41ecfc5e" gracePeriod=25 Oct 30 02:16:18.879731 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:18.879685 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b6b7fc-546d4d784c-6cjjd" podUID="9061008c-04d0-42d9-96ae-a8ebe16c54b7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:16:18.879731 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:18.879701 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b6b7fc-68dff5d4d-295gs" podUID="a36fc050-879a-4182-ac44-6d3ad2ff189e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:16:18.907560 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:18.907513 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-657fba-666fc89fb4-tc5q4"] Oct 30 02:16:18.907803 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:18.907754 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-657fba-666fc89fb4-tc5q4" podUID="d5b94b56-1785-45b7-ae04-3afb97ace8a3" containerName="maas-api" containerID="cri-o://61ee90e9490812fe7a7868fe3df1b338024c735a94f871c1de3d4d94738736dd" gracePeriod=30 Oct 30 02:16:19.178559 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:19.178156 6221 generic.go:356] "Generic (PLEG): container finished" podID="d5b94b56-1785-45b7-ae04-3afb97ace8a3" containerID="61ee90e9490812fe7a7868fe3df1b338024c735a94f871c1de3d4d94738736dd" exitCode=0 Oct 30 02:16:19.178559 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:19.178415 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-657fba-666fc89fb4-tc5q4" event={"ID":"d5b94b56-1785-45b7-ae04-3afb97ace8a3","Type":"ContainerDied","Data":"61ee90e9490812fe7a7868fe3df1b338024c735a94f871c1de3d4d94738736dd"} Oct 30 02:16:19.234325 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:19.234295 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-657fba-666fc89fb4-tc5q4" Oct 30 02:16:19.326029 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:19.325991 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5b94b56-1785-45b7-ae04-3afb97ace8a3-kube-api-access-l2b8r\" (UniqueName: \"kubernetes.io/projected/d5b94b56-1785-45b7-ae04-3afb97ace8a3-kube-api-access-l2b8r\") pod \"d5b94b56-1785-45b7-ae04-3afb97ace8a3\" (UID: \"d5b94b56-1785-45b7-ae04-3afb97ace8a3\") " Oct 30 02:16:19.326599 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:19.326569 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5b94b56-1785-45b7-ae04-3afb97ace8a3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5b94b56-1785-45b7-ae04-3afb97ace8a3-metrics-certs\") pod \"d5b94b56-1785-45b7-ae04-3afb97ace8a3\" (UID: \"d5b94b56-1785-45b7-ae04-3afb97ace8a3\") " Oct 30 02:16:19.326793 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:19.326774 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5b94b56-1785-45b7-ae04-3afb97ace8a3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5b94b56-1785-45b7-ae04-3afb97ace8a3-openshift-service-ca\") pod \"d5b94b56-1785-45b7-ae04-3afb97ace8a3\" (UID: \"d5b94b56-1785-45b7-ae04-3afb97ace8a3\") " Oct 30 02:16:19.326987 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:19.326970 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5b94b56-1785-45b7-ae04-3afb97ace8a3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5b94b56-1785-45b7-ae04-3afb97ace8a3-maas-api-tls\") pod \"d5b94b56-1785-45b7-ae04-3afb97ace8a3\" (UID: \"d5b94b56-1785-45b7-ae04-3afb97ace8a3\") " Oct 30 02:16:19.346521 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:19.341709 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5b94b56-1785-45b7-ae04-3afb97ace8a3-maas-api-tls" pod "d5b94b56-1785-45b7-ae04-3afb97ace8a3" (UID: "d5b94b56-1785-45b7-ae04-3afb97ace8a3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:16:19.346521 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:19.345755 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5b94b56-1785-45b7-ae04-3afb97ace8a3-openshift-service-ca" pod "d5b94b56-1785-45b7-ae04-3afb97ace8a3" (UID: "d5b94b56-1785-45b7-ae04-3afb97ace8a3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:16:19.352950 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:19.350398 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5b94b56-1785-45b7-ae04-3afb97ace8a3-kube-api-access-l2b8r" pod "d5b94b56-1785-45b7-ae04-3afb97ace8a3" (UID: "d5b94b56-1785-45b7-ae04-3afb97ace8a3"). InnerVolumeSpecName "kube-api-access-l2b8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:16:19.376262 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:19.376012 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5b94b56-1785-45b7-ae04-3afb97ace8a3-metrics-certs" pod "d5b94b56-1785-45b7-ae04-3afb97ace8a3" (UID: "d5b94b56-1785-45b7-ae04-3afb97ace8a3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:16:19.429739 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:19.429576 6221 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5b94b56-1785-45b7-ae04-3afb97ace8a3-metrics-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:16:19.429739 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:19.429619 6221 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5b94b56-1785-45b7-ae04-3afb97ace8a3-openshift-service-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:16:19.429739 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:19.429634 6221 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5b94b56-1785-45b7-ae04-3afb97ace8a3-maas-api-tls\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:16:19.429739 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:19.429646 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2b8r\" (UniqueName: \"kubernetes.io/projected/d5b94b56-1785-45b7-ae04-3afb97ace8a3-kube-api-access-l2b8r\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:16:20.186464 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:20.186386 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-657fba-666fc89fb4-tc5q4" Oct 30 02:16:20.186902 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:20.186391 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-657fba-666fc89fb4-tc5q4" event={"ID":"d5b94b56-1785-45b7-ae04-3afb97ace8a3","Type":"ContainerDied","Data":"20b667b9456903413f99396eb274a1d94f7fd146cc42faa81980dba047e50346"} Oct 30 02:16:20.186902 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:20.186527 6221 scope.go:122] "RemoveContainer" containerID="61ee90e9490812fe7a7868fe3df1b338024c735a94f871c1de3d4d94738736dd" Oct 30 02:16:20.242943 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:20.242911 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-657fba-666fc89fb4-tc5q4"] Oct 30 02:16:20.255408 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:20.255376 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-657fba-666fc89fb4-tc5q4"] Oct 30 02:16:20.548992 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:20.548958 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5b94b56-1785-45b7-ae04-3afb97ace8a3" path="/var/lib/kubelet/pods/d5b94b56-1785-45b7-ae04-3afb97ace8a3/volumes" Oct 30 02:16:21.518274 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:16:21.518228 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 02:16:27.996182 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:27.996144 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-aaf045-64c758fc8b-jvf8m"] Oct 30 02:16:28.026870 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:28.026835 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-aaf045-5f8df979f9-mmvp9"] Oct 30 02:16:28.879185 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:28.879146 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b6b7fc-68dff5d4d-295gs" podUID="a36fc050-879a-4182-ac44-6d3ad2ff189e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:16:28.879365 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:28.879157 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b6b7fc-546d4d784c-6cjjd" podUID="9061008c-04d0-42d9-96ae-a8ebe16c54b7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:16:32.996871 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:32.996757 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-aaf045-64c758fc8b-jvf8m" podUID="c4ed9612-c80c-4768-a30d-5d43f2868e89" containerName="payload-processing" containerID="cri-o://309c549461f74a124d595f25e8ece6f9c436133ad7318cfc9479c21231ce2b0c" gracePeriod=25 Oct 30 02:16:33.028129 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:33.028075 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aaf045-5f8df979f9-mmvp9" podUID="decbaf90-f664-408c-a7df-b7f2667f399b" containerName="payload-pre-processing" containerID="cri-o://a1bbbc5d25f508e0b330394313f3c087193f109f8321c004a794d241a5aac2bc" gracePeriod=25 Oct 30 02:16:33.267464 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:33.267402 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-aaf045-64c758fc8b-jvf8m" Oct 30 02:16:33.312043 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:33.312011 6221 generic.go:356] "Generic (PLEG): container finished" podID="c4ed9612-c80c-4768-a30d-5d43f2868e89" containerID="309c549461f74a124d595f25e8ece6f9c436133ad7318cfc9479c21231ce2b0c" exitCode=0 Oct 30 02:16:33.312220 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:33.312038 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-aaf045-64c758fc8b-jvf8m" event={"ID":"c4ed9612-c80c-4768-a30d-5d43f2868e89","Type":"ContainerDied","Data":"309c549461f74a124d595f25e8ece6f9c436133ad7318cfc9479c21231ce2b0c"} Oct 30 02:16:33.312220 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:33.312057 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-aaf045-64c758fc8b-jvf8m" Oct 30 02:16:33.312220 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:33.312125 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-aaf045-64c758fc8b-jvf8m" event={"ID":"c4ed9612-c80c-4768-a30d-5d43f2868e89","Type":"ContainerDied","Data":"2d1507d518694418a1357799bf9f498a6aa167f96d7ecc359ae19ab11b85dbb5"} Oct 30 02:16:33.312220 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:33.312155 6221 scope.go:122] "RemoveContainer" containerID="309c549461f74a124d595f25e8ece6f9c436133ad7318cfc9479c21231ce2b0c" Oct 30 02:16:33.323805 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:33.323780 6221 scope.go:122] "RemoveContainer" containerID="309c549461f74a124d595f25e8ece6f9c436133ad7318cfc9479c21231ce2b0c" Oct 30 02:16:33.324106 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:16:33.324080 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"309c549461f74a124d595f25e8ece6f9c436133ad7318cfc9479c21231ce2b0c\": container with ID starting with 309c549461f74a124d595f25e8ece6f9c436133ad7318cfc9479c21231ce2b0c not found: ID does not exist" containerID="309c549461f74a124d595f25e8ece6f9c436133ad7318cfc9479c21231ce2b0c" Oct 30 02:16:33.324178 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:33.324122 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"309c549461f74a124d595f25e8ece6f9c436133ad7318cfc9479c21231ce2b0c"} err="failed to get container status \"309c549461f74a124d595f25e8ece6f9c436133ad7318cfc9479c21231ce2b0c\": rpc error: code = NotFound desc = could not find container \"309c549461f74a124d595f25e8ece6f9c436133ad7318cfc9479c21231ce2b0c\": container with ID starting with 309c549461f74a124d595f25e8ece6f9c436133ad7318cfc9479c21231ce2b0c not found: ID does not exist" Oct 30 02:16:33.358490 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:33.358455 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4ed9612-c80c-4768-a30d-5d43f2868e89-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4ed9612-c80c-4768-a30d-5d43f2868e89-plugins-config-volume\") pod \"c4ed9612-c80c-4768-a30d-5d43f2868e89\" (UID: \"c4ed9612-c80c-4768-a30d-5d43f2868e89\") " Oct 30 02:16:33.358654 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:33.358512 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4ed9612-c80c-4768-a30d-5d43f2868e89-kube-api-access-rq4fl\" (UniqueName: \"kubernetes.io/projected/c4ed9612-c80c-4768-a30d-5d43f2868e89-kube-api-access-rq4fl\") pod \"c4ed9612-c80c-4768-a30d-5d43f2868e89\" (UID: \"c4ed9612-c80c-4768-a30d-5d43f2868e89\") " Oct 30 02:16:33.358938 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:33.358914 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4ed9612-c80c-4768-a30d-5d43f2868e89-plugins-config-volume" pod "c4ed9612-c80c-4768-a30d-5d43f2868e89" (UID: "c4ed9612-c80c-4768-a30d-5d43f2868e89"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:16:33.359187 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:33.359165 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4ed9612-c80c-4768-a30d-5d43f2868e89-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:16:33.360781 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:33.360755 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4ed9612-c80c-4768-a30d-5d43f2868e89-kube-api-access-rq4fl" pod "c4ed9612-c80c-4768-a30d-5d43f2868e89" (UID: "c4ed9612-c80c-4768-a30d-5d43f2868e89"). InnerVolumeSpecName "kube-api-access-rq4fl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:16:33.460247 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:33.460219 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rq4fl\" (UniqueName: \"kubernetes.io/projected/c4ed9612-c80c-4768-a30d-5d43f2868e89-kube-api-access-rq4fl\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:16:33.659409 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:33.659376 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-aaf045-64c758fc8b-jvf8m"] Oct 30 02:16:33.663810 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:33.663776 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-aaf045-64c758fc8b-jvf8m"] Oct 30 02:16:34.548220 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:34.548185 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4ed9612-c80c-4768-a30d-5d43f2868e89" path="/var/lib/kubelet/pods/c4ed9612-c80c-4768-a30d-5d43f2868e89/volumes" Oct 30 02:16:36.003526 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:36.003490 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aaf045-5f8df979f9-mmvp9" podUID="decbaf90-f664-408c-a7df-b7f2667f399b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:16:36.015168 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:36.015142 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b6b7fc-68dff5d4d-295gs" Oct 30 02:16:36.037373 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:36.037352 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b6b7fc-546d4d784c-6cjjd" Oct 30 02:16:36.103416 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:36.103381 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9061008c-04d0-42d9-96ae-a8ebe16c54b7-kube-api-access-6ccjd\" (UniqueName: \"kubernetes.io/projected/9061008c-04d0-42d9-96ae-a8ebe16c54b7-kube-api-access-6ccjd\") pod \"9061008c-04d0-42d9-96ae-a8ebe16c54b7\" (UID: \"9061008c-04d0-42d9-96ae-a8ebe16c54b7\") " Oct 30 02:16:36.104141 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:36.104117 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9061008c-04d0-42d9-96ae-a8ebe16c54b7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9061008c-04d0-42d9-96ae-a8ebe16c54b7-plugins-config-volume\") pod \"9061008c-04d0-42d9-96ae-a8ebe16c54b7\" (UID: \"9061008c-04d0-42d9-96ae-a8ebe16c54b7\") " Oct 30 02:16:36.105629 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:36.104656 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9061008c-04d0-42d9-96ae-a8ebe16c54b7-plugins-config-volume" pod "9061008c-04d0-42d9-96ae-a8ebe16c54b7" (UID: "9061008c-04d0-42d9-96ae-a8ebe16c54b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:16:36.105752 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:36.105603 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a36fc050-879a-4182-ac44-6d3ad2ff189e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a36fc050-879a-4182-ac44-6d3ad2ff189e-plugins-config-volume\") pod \"a36fc050-879a-4182-ac44-6d3ad2ff189e\" (UID: \"a36fc050-879a-4182-ac44-6d3ad2ff189e\") " Oct 30 02:16:36.105752 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:36.105686 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a36fc050-879a-4182-ac44-6d3ad2ff189e-kube-api-access-fvm5m\" (UniqueName: \"kubernetes.io/projected/a36fc050-879a-4182-ac44-6d3ad2ff189e-kube-api-access-fvm5m\") pod \"a36fc050-879a-4182-ac44-6d3ad2ff189e\" (UID: \"a36fc050-879a-4182-ac44-6d3ad2ff189e\") " Oct 30 02:16:36.106052 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:36.106025 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a36fc050-879a-4182-ac44-6d3ad2ff189e-plugins-config-volume" pod "a36fc050-879a-4182-ac44-6d3ad2ff189e" (UID: "a36fc050-879a-4182-ac44-6d3ad2ff189e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:16:36.106472 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:36.106442 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9061008c-04d0-42d9-96ae-a8ebe16c54b7-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:16:36.106623 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:36.106475 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a36fc050-879a-4182-ac44-6d3ad2ff189e-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:16:36.108075 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:36.108043 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9061008c-04d0-42d9-96ae-a8ebe16c54b7-kube-api-access-6ccjd" pod "9061008c-04d0-42d9-96ae-a8ebe16c54b7" (UID: "9061008c-04d0-42d9-96ae-a8ebe16c54b7"). InnerVolumeSpecName "kube-api-access-6ccjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:16:36.108169 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:36.108093 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a36fc050-879a-4182-ac44-6d3ad2ff189e-kube-api-access-fvm5m" pod "a36fc050-879a-4182-ac44-6d3ad2ff189e" (UID: "a36fc050-879a-4182-ac44-6d3ad2ff189e"). InnerVolumeSpecName "kube-api-access-fvm5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:16:36.208194 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:36.208118 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvm5m\" (UniqueName: \"kubernetes.io/projected/a36fc050-879a-4182-ac44-6d3ad2ff189e-kube-api-access-fvm5m\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:16:36.208194 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:36.208147 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ccjd\" (UniqueName: \"kubernetes.io/projected/9061008c-04d0-42d9-96ae-a8ebe16c54b7-kube-api-access-6ccjd\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:16:36.336654 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:36.336619 6221 generic.go:356] "Generic (PLEG): container finished" podID="9061008c-04d0-42d9-96ae-a8ebe16c54b7" containerID="76444066abf01df9d4275cd0ef6c2264589cb0712c1360fe7b6d9aff41ecfc5e" exitCode=0 Oct 30 02:16:36.336842 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:36.336668 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b6b7fc-546d4d784c-6cjjd" Oct 30 02:16:36.336842 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:36.336703 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b6b7fc-546d4d784c-6cjjd" event={"ID":"9061008c-04d0-42d9-96ae-a8ebe16c54b7","Type":"ContainerDied","Data":"76444066abf01df9d4275cd0ef6c2264589cb0712c1360fe7b6d9aff41ecfc5e"} Oct 30 02:16:36.336842 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:36.336762 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b6b7fc-546d4d784c-6cjjd" event={"ID":"9061008c-04d0-42d9-96ae-a8ebe16c54b7","Type":"ContainerDied","Data":"46b4ab5bbd837a0303bf3e5882c90314828809b5ad3469b1c57dd27952530154"} Oct 30 02:16:36.336842 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:36.336786 6221 scope.go:122] "RemoveContainer" containerID="76444066abf01df9d4275cd0ef6c2264589cb0712c1360fe7b6d9aff41ecfc5e" Oct 30 02:16:36.338564 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:36.338521 6221 generic.go:356] "Generic (PLEG): container finished" podID="a36fc050-879a-4182-ac44-6d3ad2ff189e" containerID="8a81ae121384c2eae6168aa3a7649f9dfc706763a4a454c9c2e956a5d966cd44" exitCode=0 Oct 30 02:16:36.338681 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:36.338579 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b6b7fc-68dff5d4d-295gs" event={"ID":"a36fc050-879a-4182-ac44-6d3ad2ff189e","Type":"ContainerDied","Data":"8a81ae121384c2eae6168aa3a7649f9dfc706763a4a454c9c2e956a5d966cd44"} Oct 30 02:16:36.338681 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:36.338605 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b6b7fc-68dff5d4d-295gs" Oct 30 02:16:36.338762 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:36.338618 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-b6b7fc-68dff5d4d-295gs" event={"ID":"a36fc050-879a-4182-ac44-6d3ad2ff189e","Type":"ContainerDied","Data":"5423a8546089615a9991bfdeab5345de774e5012104e25f4622feec1c58b19dc"} Oct 30 02:16:36.348092 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:36.348069 6221 scope.go:122] "RemoveContainer" containerID="76444066abf01df9d4275cd0ef6c2264589cb0712c1360fe7b6d9aff41ecfc5e" Oct 30 02:16:36.348366 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:16:36.348349 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76444066abf01df9d4275cd0ef6c2264589cb0712c1360fe7b6d9aff41ecfc5e\": container with ID starting with 76444066abf01df9d4275cd0ef6c2264589cb0712c1360fe7b6d9aff41ecfc5e not found: ID does not exist" containerID="76444066abf01df9d4275cd0ef6c2264589cb0712c1360fe7b6d9aff41ecfc5e" Oct 30 02:16:36.348431 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:36.348372 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76444066abf01df9d4275cd0ef6c2264589cb0712c1360fe7b6d9aff41ecfc5e"} err="failed to get container status \"76444066abf01df9d4275cd0ef6c2264589cb0712c1360fe7b6d9aff41ecfc5e\": rpc error: code = NotFound desc = could not find container \"76444066abf01df9d4275cd0ef6c2264589cb0712c1360fe7b6d9aff41ecfc5e\": container with ID starting with 76444066abf01df9d4275cd0ef6c2264589cb0712c1360fe7b6d9aff41ecfc5e not found: ID does not exist" Oct 30 02:16:36.348431 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:36.348389 6221 scope.go:122] "RemoveContainer" containerID="8a81ae121384c2eae6168aa3a7649f9dfc706763a4a454c9c2e956a5d966cd44" Oct 30 02:16:36.357877 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:36.357859 6221 scope.go:122] "RemoveContainer" containerID="8a81ae121384c2eae6168aa3a7649f9dfc706763a4a454c9c2e956a5d966cd44" Oct 30 02:16:36.358103 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:16:36.358086 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a81ae121384c2eae6168aa3a7649f9dfc706763a4a454c9c2e956a5d966cd44\": container with ID starting with 8a81ae121384c2eae6168aa3a7649f9dfc706763a4a454c9c2e956a5d966cd44 not found: ID does not exist" containerID="8a81ae121384c2eae6168aa3a7649f9dfc706763a4a454c9c2e956a5d966cd44" Oct 30 02:16:36.358154 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:36.358109 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a81ae121384c2eae6168aa3a7649f9dfc706763a4a454c9c2e956a5d966cd44"} err="failed to get container status \"8a81ae121384c2eae6168aa3a7649f9dfc706763a4a454c9c2e956a5d966cd44\": rpc error: code = NotFound desc = could not find container \"8a81ae121384c2eae6168aa3a7649f9dfc706763a4a454c9c2e956a5d966cd44\": container with ID starting with 8a81ae121384c2eae6168aa3a7649f9dfc706763a4a454c9c2e956a5d966cd44 not found: ID does not exist" Oct 30 02:16:36.391096 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:36.391060 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b6b7fc-68dff5d4d-295gs"] Oct 30 02:16:36.403649 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:36.403623 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-b6b7fc-68dff5d4d-295gs"] Oct 30 02:16:36.437646 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:36.437614 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b6b7fc-546d4d784c-6cjjd"] Oct 30 02:16:36.441957 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:36.441928 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-b6b7fc-546d4d784c-6cjjd"] Oct 30 02:16:36.548156 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:36.548123 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9061008c-04d0-42d9-96ae-a8ebe16c54b7" path="/var/lib/kubelet/pods/9061008c-04d0-42d9-96ae-a8ebe16c54b7/volumes" Oct 30 02:16:36.548567 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:36.548553 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a36fc050-879a-4182-ac44-6d3ad2ff189e" path="/var/lib/kubelet/pods/a36fc050-879a-4182-ac44-6d3ad2ff189e/volumes" Oct 30 02:16:46.004216 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:46.004167 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aaf045-5f8df979f9-mmvp9" podUID="decbaf90-f664-408c-a7df-b7f2667f399b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:16:46.319381 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:46.319345 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f89697/test-model-f89697-kserve-676c47d5c-ntkcd"] Oct 30 02:16:51.495295 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:16:51.495256 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 02:16:53.214958 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:53.214904 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m"] Oct 30 02:16:53.215469 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:53.215211 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" podUID="a5227b83-d834-4276-bd61-27e1251a0708" containerName="istio-proxy" containerID="cri-o://3a361174a51616ed91cd94aa605449f812dcc9e77a996937b6a1293f691464a0" gracePeriod=30 Oct 30 02:16:53.470688 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:53.470315 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aaf045-5f8df979f9-mmvp9" Oct 30 02:16:53.594269 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:53.587134 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/decbaf90-f664-408c-a7df-b7f2667f399b-kube-api-access-w77hn\" (UniqueName: \"kubernetes.io/projected/decbaf90-f664-408c-a7df-b7f2667f399b-kube-api-access-w77hn\") pod \"decbaf90-f664-408c-a7df-b7f2667f399b\" (UID: \"decbaf90-f664-408c-a7df-b7f2667f399b\") " Oct 30 02:16:53.594269 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:53.587225 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/decbaf90-f664-408c-a7df-b7f2667f399b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/decbaf90-f664-408c-a7df-b7f2667f399b-plugins-config-volume\") pod \"decbaf90-f664-408c-a7df-b7f2667f399b\" (UID: \"decbaf90-f664-408c-a7df-b7f2667f399b\") " Oct 30 02:16:53.594269 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:53.588570 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/decbaf90-f664-408c-a7df-b7f2667f399b-plugins-config-volume" pod "decbaf90-f664-408c-a7df-b7f2667f399b" (UID: "decbaf90-f664-408c-a7df-b7f2667f399b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:16:53.594269 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:53.589224 6221 generic.go:356] "Generic (PLEG): container finished" podID="decbaf90-f664-408c-a7df-b7f2667f399b" containerID="a1bbbc5d25f508e0b330394313f3c087193f109f8321c004a794d241a5aac2bc" exitCode=0 Oct 30 02:16:53.594269 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:53.589334 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aaf045-5f8df979f9-mmvp9" event={"ID":"decbaf90-f664-408c-a7df-b7f2667f399b","Type":"ContainerDied","Data":"a1bbbc5d25f508e0b330394313f3c087193f109f8321c004a794d241a5aac2bc"} Oct 30 02:16:53.594269 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:53.589371 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aaf045-5f8df979f9-mmvp9" event={"ID":"decbaf90-f664-408c-a7df-b7f2667f399b","Type":"ContainerDied","Data":"c39249ccb90530be985013a3bc07033686c2ba02e096bc94f0b4d29b4b4aee60"} Oct 30 02:16:53.594269 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:53.589392 6221 scope.go:122] "RemoveContainer" containerID="a1bbbc5d25f508e0b330394313f3c087193f109f8321c004a794d241a5aac2bc" Oct 30 02:16:53.594269 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:53.589549 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aaf045-5f8df979f9-mmvp9" Oct 30 02:16:53.599628 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:53.595324 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/decbaf90-f664-408c-a7df-b7f2667f399b-kube-api-access-w77hn" pod "decbaf90-f664-408c-a7df-b7f2667f399b" (UID: "decbaf90-f664-408c-a7df-b7f2667f399b"). InnerVolumeSpecName "kube-api-access-w77hn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:16:53.600860 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:53.600833 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz"] Oct 30 02:16:53.601063 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:53.601037 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" podUID="fd152ecb-4d3c-494c-83c1-182f018a4db1" containerName="istio-proxy" containerID="cri-o://ea107ead421b1caa03cf70f9c55113a0a0667cb05b0215b77a8b54dc3edcf5d7" gracePeriod=30 Oct 30 02:16:53.610470 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:53.610449 6221 scope.go:122] "RemoveContainer" containerID="a1bbbc5d25f508e0b330394313f3c087193f109f8321c004a794d241a5aac2bc" Oct 30 02:16:53.610825 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:16:53.610797 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1bbbc5d25f508e0b330394313f3c087193f109f8321c004a794d241a5aac2bc\": container with ID starting with a1bbbc5d25f508e0b330394313f3c087193f109f8321c004a794d241a5aac2bc not found: ID does not exist" containerID="a1bbbc5d25f508e0b330394313f3c087193f109f8321c004a794d241a5aac2bc" Oct 30 02:16:53.610917 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:53.610837 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1bbbc5d25f508e0b330394313f3c087193f109f8321c004a794d241a5aac2bc"} err="failed to get container status \"a1bbbc5d25f508e0b330394313f3c087193f109f8321c004a794d241a5aac2bc\": rpc error: code = NotFound desc = could not find container \"a1bbbc5d25f508e0b330394313f3c087193f109f8321c004a794d241a5aac2bc\": container with ID starting with a1bbbc5d25f508e0b330394313f3c087193f109f8321c004a794d241a5aac2bc not found: ID does not exist" Oct 30 02:16:53.690357 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:53.690275 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w77hn\" (UniqueName: \"kubernetes.io/projected/decbaf90-f664-408c-a7df-b7f2667f399b-kube-api-access-w77hn\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:16:53.690357 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:53.690306 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/decbaf90-f664-408c-a7df-b7f2667f399b-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:16:53.745727 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:53.745689 6221 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:16:53.746005 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:53.745978 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" podUID="fd152ecb-4d3c-494c-83c1-182f018a4db1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:16:53.944472 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:53.944395 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-aaf045-5f8df979f9-mmvp9"] Oct 30 02:16:53.947420 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:53.947393 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-aaf045-5f8df979f9-mmvp9"] Oct 30 02:16:54.549372 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:54.549333 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="decbaf90-f664-408c-a7df-b7f2667f399b" path="/var/lib/kubelet/pods/decbaf90-f664-408c-a7df-b7f2667f399b/volumes" Oct 30 02:16:58.441714 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.441344 6221 scope.go:122] "RemoveContainer" containerID="81581d75d759bd496f2d21225e362465f79d6d3968703b85be3cc502745b1332" Oct 30 02:16:58.507568 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.507543 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" Oct 30 02:16:58.630072 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.630009 6221 generic.go:356] "Generic (PLEG): container finished" podID="a5227b83-d834-4276-bd61-27e1251a0708" containerID="3a361174a51616ed91cd94aa605449f812dcc9e77a996937b6a1293f691464a0" exitCode=0 Oct 30 02:16:58.630165 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.630091 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" Oct 30 02:16:58.630165 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.630097 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" event={"ID":"a5227b83-d834-4276-bd61-27e1251a0708","Type":"ContainerDied","Data":"3a361174a51616ed91cd94aa605449f812dcc9e77a996937b6a1293f691464a0"} Oct 30 02:16:58.630285 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.630169 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m" event={"ID":"a5227b83-d834-4276-bd61-27e1251a0708","Type":"ContainerDied","Data":"5d370e8c37b3fd8fe1e9254e24190df98959d9647cb425d8fcc4de683d1ebb88"} Oct 30 02:16:58.630285 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.630205 6221 scope.go:122] "RemoveContainer" containerID="3a361174a51616ed91cd94aa605449f812dcc9e77a996937b6a1293f691464a0" Oct 30 02:16:58.666602 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.666575 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5227b83-d834-4276-bd61-27e1251a0708-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5227b83-d834-4276-bd61-27e1251a0708-istio-podinfo\") pod \"a5227b83-d834-4276-bd61-27e1251a0708\" (UID: \"a5227b83-d834-4276-bd61-27e1251a0708\") " Oct 30 02:16:58.666801 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.666611 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5227b83-d834-4276-bd61-27e1251a0708-istio-token\" (UniqueName: \"kubernetes.io/projected/a5227b83-d834-4276-bd61-27e1251a0708-istio-token\") pod \"a5227b83-d834-4276-bd61-27e1251a0708\" (UID: \"a5227b83-d834-4276-bd61-27e1251a0708\") " Oct 30 02:16:58.666801 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.666640 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5227b83-d834-4276-bd61-27e1251a0708-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5227b83-d834-4276-bd61-27e1251a0708-istio-envoy\") pod \"a5227b83-d834-4276-bd61-27e1251a0708\" (UID: \"a5227b83-d834-4276-bd61-27e1251a0708\") " Oct 30 02:16:58.666801 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.666734 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5227b83-d834-4276-bd61-27e1251a0708-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5227b83-d834-4276-bd61-27e1251a0708-istio-data\") pod \"a5227b83-d834-4276-bd61-27e1251a0708\" (UID: \"a5227b83-d834-4276-bd61-27e1251a0708\") " Oct 30 02:16:58.666801 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.666767 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5227b83-d834-4276-bd61-27e1251a0708-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5227b83-d834-4276-bd61-27e1251a0708-workload-socket\") pod \"a5227b83-d834-4276-bd61-27e1251a0708\" (UID: \"a5227b83-d834-4276-bd61-27e1251a0708\") " Oct 30 02:16:58.667059 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.666804 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5227b83-d834-4276-bd61-27e1251a0708-kube-api-access-s5pq8\" (UniqueName: \"kubernetes.io/projected/a5227b83-d834-4276-bd61-27e1251a0708-kube-api-access-s5pq8\") pod \"a5227b83-d834-4276-bd61-27e1251a0708\" (UID: \"a5227b83-d834-4276-bd61-27e1251a0708\") " Oct 30 02:16:58.667059 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.666860 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5227b83-d834-4276-bd61-27e1251a0708-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5227b83-d834-4276-bd61-27e1251a0708-workload-certs\") pod \"a5227b83-d834-4276-bd61-27e1251a0708\" (UID: \"a5227b83-d834-4276-bd61-27e1251a0708\") " Oct 30 02:16:58.667059 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.666914 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5227b83-d834-4276-bd61-27e1251a0708-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5227b83-d834-4276-bd61-27e1251a0708-istiod-ca-cert\") pod \"a5227b83-d834-4276-bd61-27e1251a0708\" (UID: \"a5227b83-d834-4276-bd61-27e1251a0708\") " Oct 30 02:16:58.667059 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.667030 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5227b83-d834-4276-bd61-27e1251a0708-istio-data" pod "a5227b83-d834-4276-bd61-27e1251a0708" (UID: "a5227b83-d834-4276-bd61-27e1251a0708"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:16:58.667289 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.667186 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5227b83-d834-4276-bd61-27e1251a0708-workload-socket" pod "a5227b83-d834-4276-bd61-27e1251a0708" (UID: "a5227b83-d834-4276-bd61-27e1251a0708"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:16:58.667289 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.667231 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5227b83-d834-4276-bd61-27e1251a0708-workload-certs" pod "a5227b83-d834-4276-bd61-27e1251a0708" (UID: "a5227b83-d834-4276-bd61-27e1251a0708"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:16:58.667289 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.667256 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5227b83-d834-4276-bd61-27e1251a0708-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5227b83-d834-4276-bd61-27e1251a0708-credential-socket\") pod \"a5227b83-d834-4276-bd61-27e1251a0708\" (UID: \"a5227b83-d834-4276-bd61-27e1251a0708\") " Oct 30 02:16:58.667471 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.667443 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5227b83-d834-4276-bd61-27e1251a0708-credential-socket" pod "a5227b83-d834-4276-bd61-27e1251a0708" (UID: "a5227b83-d834-4276-bd61-27e1251a0708"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:16:58.667471 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.667375 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5227b83-d834-4276-bd61-27e1251a0708-istiod-ca-cert" pod "a5227b83-d834-4276-bd61-27e1251a0708" (UID: "a5227b83-d834-4276-bd61-27e1251a0708"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:16:58.668147 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.668123 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5227b83-d834-4276-bd61-27e1251a0708-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:16:58.668276 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.668151 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5227b83-d834-4276-bd61-27e1251a0708-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:16:58.668276 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.668167 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5227b83-d834-4276-bd61-27e1251a0708-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:16:58.668276 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.668182 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5227b83-d834-4276-bd61-27e1251a0708-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:16:58.668276 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.668196 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5227b83-d834-4276-bd61-27e1251a0708-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:16:58.669168 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.669149 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5227b83-d834-4276-bd61-27e1251a0708-istio-envoy" pod "a5227b83-d834-4276-bd61-27e1251a0708" (UID: "a5227b83-d834-4276-bd61-27e1251a0708"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:16:58.669263 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.669221 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5227b83-d834-4276-bd61-27e1251a0708-istio-token" pod "a5227b83-d834-4276-bd61-27e1251a0708" (UID: "a5227b83-d834-4276-bd61-27e1251a0708"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:16:58.669853 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.669831 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5227b83-d834-4276-bd61-27e1251a0708-istio-podinfo" pod "a5227b83-d834-4276-bd61-27e1251a0708" (UID: "a5227b83-d834-4276-bd61-27e1251a0708"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:16:58.669937 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.669869 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5227b83-d834-4276-bd61-27e1251a0708-kube-api-access-s5pq8" pod "a5227b83-d834-4276-bd61-27e1251a0708" (UID: "a5227b83-d834-4276-bd61-27e1251a0708"). InnerVolumeSpecName "kube-api-access-s5pq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:16:58.716841 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.716818 6221 scope.go:122] "RemoveContainer" containerID="3a361174a51616ed91cd94aa605449f812dcc9e77a996937b6a1293f691464a0" Oct 30 02:16:58.717190 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:16:58.717165 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a361174a51616ed91cd94aa605449f812dcc9e77a996937b6a1293f691464a0\": container with ID starting with 3a361174a51616ed91cd94aa605449f812dcc9e77a996937b6a1293f691464a0 not found: ID does not exist" containerID="3a361174a51616ed91cd94aa605449f812dcc9e77a996937b6a1293f691464a0" Oct 30 02:16:58.717301 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.717198 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a361174a51616ed91cd94aa605449f812dcc9e77a996937b6a1293f691464a0"} err="failed to get container status \"3a361174a51616ed91cd94aa605449f812dcc9e77a996937b6a1293f691464a0\": rpc error: code = NotFound desc = could not find container \"3a361174a51616ed91cd94aa605449f812dcc9e77a996937b6a1293f691464a0\": container with ID starting with 3a361174a51616ed91cd94aa605449f812dcc9e77a996937b6a1293f691464a0 not found: ID does not exist" Oct 30 02:16:58.769632 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.769543 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5227b83-d834-4276-bd61-27e1251a0708-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:16:58.769632 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.769579 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5227b83-d834-4276-bd61-27e1251a0708-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:16:58.769632 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.769588 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5227b83-d834-4276-bd61-27e1251a0708-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:16:58.769632 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.769597 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5pq8\" (UniqueName: \"kubernetes.io/projected/a5227b83-d834-4276-bd61-27e1251a0708-kube-api-access-s5pq8\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:16:58.850263 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.850240 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" Oct 30 02:16:58.971253 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.971222 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd152ecb-4d3c-494c-83c1-182f018a4db1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd152ecb-4d3c-494c-83c1-182f018a4db1-credential-socket\") pod \"fd152ecb-4d3c-494c-83c1-182f018a4db1\" (UID: \"fd152ecb-4d3c-494c-83c1-182f018a4db1\") " Oct 30 02:16:58.971443 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.971354 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd152ecb-4d3c-494c-83c1-182f018a4db1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd152ecb-4d3c-494c-83c1-182f018a4db1-workload-socket\") pod \"fd152ecb-4d3c-494c-83c1-182f018a4db1\" (UID: \"fd152ecb-4d3c-494c-83c1-182f018a4db1\") " Oct 30 02:16:58.971443 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.971387 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd152ecb-4d3c-494c-83c1-182f018a4db1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd152ecb-4d3c-494c-83c1-182f018a4db1-istio-data\") pod \"fd152ecb-4d3c-494c-83c1-182f018a4db1\" (UID: \"fd152ecb-4d3c-494c-83c1-182f018a4db1\") " Oct 30 02:16:58.971443 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.971429 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd152ecb-4d3c-494c-83c1-182f018a4db1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd152ecb-4d3c-494c-83c1-182f018a4db1-istiod-ca-cert\") pod \"fd152ecb-4d3c-494c-83c1-182f018a4db1\" (UID: \"fd152ecb-4d3c-494c-83c1-182f018a4db1\") " Oct 30 02:16:58.971704 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.971478 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd152ecb-4d3c-494c-83c1-182f018a4db1-credential-socket" pod "fd152ecb-4d3c-494c-83c1-182f018a4db1" (UID: "fd152ecb-4d3c-494c-83c1-182f018a4db1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:16:58.971704 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.971494 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd152ecb-4d3c-494c-83c1-182f018a4db1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd152ecb-4d3c-494c-83c1-182f018a4db1-workload-certs\") pod \"fd152ecb-4d3c-494c-83c1-182f018a4db1\" (UID: \"fd152ecb-4d3c-494c-83c1-182f018a4db1\") " Oct 30 02:16:58.971704 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.971585 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd152ecb-4d3c-494c-83c1-182f018a4db1-kube-api-access-rqm52\" (UniqueName: \"kubernetes.io/projected/fd152ecb-4d3c-494c-83c1-182f018a4db1-kube-api-access-rqm52\") pod \"fd152ecb-4d3c-494c-83c1-182f018a4db1\" (UID: \"fd152ecb-4d3c-494c-83c1-182f018a4db1\") " Oct 30 02:16:58.971704 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.971644 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd152ecb-4d3c-494c-83c1-182f018a4db1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd152ecb-4d3c-494c-83c1-182f018a4db1-istio-podinfo\") pod \"fd152ecb-4d3c-494c-83c1-182f018a4db1\" (UID: \"fd152ecb-4d3c-494c-83c1-182f018a4db1\") " Oct 30 02:16:58.971704 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.971686 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd152ecb-4d3c-494c-83c1-182f018a4db1-workload-socket" pod "fd152ecb-4d3c-494c-83c1-182f018a4db1" (UID: "fd152ecb-4d3c-494c-83c1-182f018a4db1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:16:58.971971 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.971702 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd152ecb-4d3c-494c-83c1-182f018a4db1-istio-data" pod "fd152ecb-4d3c-494c-83c1-182f018a4db1" (UID: "fd152ecb-4d3c-494c-83c1-182f018a4db1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:16:58.971971 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.971707 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd152ecb-4d3c-494c-83c1-182f018a4db1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd152ecb-4d3c-494c-83c1-182f018a4db1-istio-envoy\") pod \"fd152ecb-4d3c-494c-83c1-182f018a4db1\" (UID: \"fd152ecb-4d3c-494c-83c1-182f018a4db1\") " Oct 30 02:16:58.971971 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.971801 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd152ecb-4d3c-494c-83c1-182f018a4db1-workload-certs" pod "fd152ecb-4d3c-494c-83c1-182f018a4db1" (UID: "fd152ecb-4d3c-494c-83c1-182f018a4db1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:16:58.971971 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.971829 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd152ecb-4d3c-494c-83c1-182f018a4db1-istio-token\" (UniqueName: \"kubernetes.io/projected/fd152ecb-4d3c-494c-83c1-182f018a4db1-istio-token\") pod \"fd152ecb-4d3c-494c-83c1-182f018a4db1\" (UID: \"fd152ecb-4d3c-494c-83c1-182f018a4db1\") " Oct 30 02:16:58.971971 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.971851 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd152ecb-4d3c-494c-83c1-182f018a4db1-istiod-ca-cert" pod "fd152ecb-4d3c-494c-83c1-182f018a4db1" (UID: "fd152ecb-4d3c-494c-83c1-182f018a4db1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:16:58.972437 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.972409 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd152ecb-4d3c-494c-83c1-182f018a4db1-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:16:58.972509 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.972436 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd152ecb-4d3c-494c-83c1-182f018a4db1-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:16:58.972509 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.972451 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd152ecb-4d3c-494c-83c1-182f018a4db1-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:16:58.972509 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.972463 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd152ecb-4d3c-494c-83c1-182f018a4db1-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:16:58.972509 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.972476 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd152ecb-4d3c-494c-83c1-182f018a4db1-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:16:58.973799 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.973772 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd152ecb-4d3c-494c-83c1-182f018a4db1-istio-podinfo" pod "fd152ecb-4d3c-494c-83c1-182f018a4db1" (UID: "fd152ecb-4d3c-494c-83c1-182f018a4db1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:16:58.973903 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.973884 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd152ecb-4d3c-494c-83c1-182f018a4db1-istio-token" pod "fd152ecb-4d3c-494c-83c1-182f018a4db1" (UID: "fd152ecb-4d3c-494c-83c1-182f018a4db1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:16:58.974182 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.974164 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd152ecb-4d3c-494c-83c1-182f018a4db1-istio-envoy" pod "fd152ecb-4d3c-494c-83c1-182f018a4db1" (UID: "fd152ecb-4d3c-494c-83c1-182f018a4db1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:16:58.974244 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.974201 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd152ecb-4d3c-494c-83c1-182f018a4db1-kube-api-access-rqm52" pod "fd152ecb-4d3c-494c-83c1-182f018a4db1" (UID: "fd152ecb-4d3c-494c-83c1-182f018a4db1"). InnerVolumeSpecName "kube-api-access-rqm52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:16:58.980317 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.980295 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m"] Oct 30 02:16:58.985734 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:58.985712 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-aaf045-openshift-default-5cb98f8589-p226m"] Oct 30 02:16:59.073886 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:59.073857 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqm52\" (UniqueName: \"kubernetes.io/projected/fd152ecb-4d3c-494c-83c1-182f018a4db1-kube-api-access-rqm52\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:16:59.073886 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:59.073886 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd152ecb-4d3c-494c-83c1-182f018a4db1-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:16:59.074060 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:59.073899 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd152ecb-4d3c-494c-83c1-182f018a4db1-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:16:59.074060 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:59.073910 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd152ecb-4d3c-494c-83c1-182f018a4db1-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:16:59.638246 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:59.638211 6221 generic.go:356] "Generic (PLEG): container finished" podID="fd152ecb-4d3c-494c-83c1-182f018a4db1" containerID="ea107ead421b1caa03cf70f9c55113a0a0667cb05b0215b77a8b54dc3edcf5d7" exitCode=0 Oct 30 02:16:59.638750 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:59.638254 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" Oct 30 02:16:59.638750 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:59.638293 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" event={"ID":"fd152ecb-4d3c-494c-83c1-182f018a4db1","Type":"ContainerDied","Data":"ea107ead421b1caa03cf70f9c55113a0a0667cb05b0215b77a8b54dc3edcf5d7"} Oct 30 02:16:59.638750 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:59.638348 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz" event={"ID":"fd152ecb-4d3c-494c-83c1-182f018a4db1","Type":"ContainerDied","Data":"652f9af5779cb77be07e0810ceec9b352202b3854a9c6388b6f308b7e7618d12"} Oct 30 02:16:59.638750 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:59.638379 6221 scope.go:122] "RemoveContainer" containerID="ea107ead421b1caa03cf70f9c55113a0a0667cb05b0215b77a8b54dc3edcf5d7" Oct 30 02:16:59.650772 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:59.648834 6221 scope.go:122] "RemoveContainer" containerID="39aaa2dd6656f6d14ad55ea410fb92f75c423e6c041af570bed8c4a989d616b1" Oct 30 02:16:59.659203 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:59.659186 6221 scope.go:122] "RemoveContainer" containerID="ea107ead421b1caa03cf70f9c55113a0a0667cb05b0215b77a8b54dc3edcf5d7" Oct 30 02:16:59.659484 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:16:59.659464 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea107ead421b1caa03cf70f9c55113a0a0667cb05b0215b77a8b54dc3edcf5d7\": container with ID starting with ea107ead421b1caa03cf70f9c55113a0a0667cb05b0215b77a8b54dc3edcf5d7 not found: ID does not exist" containerID="ea107ead421b1caa03cf70f9c55113a0a0667cb05b0215b77a8b54dc3edcf5d7" Oct 30 02:16:59.659541 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:59.659492 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea107ead421b1caa03cf70f9c55113a0a0667cb05b0215b77a8b54dc3edcf5d7"} err="failed to get container status \"ea107ead421b1caa03cf70f9c55113a0a0667cb05b0215b77a8b54dc3edcf5d7\": rpc error: code = NotFound desc = could not find container \"ea107ead421b1caa03cf70f9c55113a0a0667cb05b0215b77a8b54dc3edcf5d7\": container with ID starting with ea107ead421b1caa03cf70f9c55113a0a0667cb05b0215b77a8b54dc3edcf5d7 not found: ID does not exist" Oct 30 02:16:59.659541 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:59.659511 6221 scope.go:122] "RemoveContainer" containerID="39aaa2dd6656f6d14ad55ea410fb92f75c423e6c041af570bed8c4a989d616b1" Oct 30 02:16:59.659787 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:16:59.659772 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39aaa2dd6656f6d14ad55ea410fb92f75c423e6c041af570bed8c4a989d616b1\": container with ID starting with 39aaa2dd6656f6d14ad55ea410fb92f75c423e6c041af570bed8c4a989d616b1 not found: ID does not exist" containerID="39aaa2dd6656f6d14ad55ea410fb92f75c423e6c041af570bed8c4a989d616b1" Oct 30 02:16:59.659842 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:59.659791 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39aaa2dd6656f6d14ad55ea410fb92f75c423e6c041af570bed8c4a989d616b1"} err="failed to get container status \"39aaa2dd6656f6d14ad55ea410fb92f75c423e6c041af570bed8c4a989d616b1\": rpc error: code = NotFound desc = could not find container \"39aaa2dd6656f6d14ad55ea410fb92f75c423e6c041af570bed8c4a989d616b1\": container with ID starting with 39aaa2dd6656f6d14ad55ea410fb92f75c423e6c041af570bed8c4a989d616b1 not found: ID does not exist" Oct 30 02:16:59.692287 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:59.692255 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz"] Oct 30 02:16:59.704643 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:16:59.704615 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b6b7fc-openshift-default-6c68b4d64c-l4dkz"] Oct 30 02:17:00.549037 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:00.549007 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5227b83-d834-4276-bd61-27e1251a0708" path="/var/lib/kubelet/pods/a5227b83-d834-4276-bd61-27e1251a0708/volumes" Oct 30 02:17:00.549519 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:00.549504 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd152ecb-4d3c-494c-83c1-182f018a4db1" path="/var/lib/kubelet/pods/fd152ecb-4d3c-494c-83c1-182f018a4db1/volumes" Oct 30 02:17:00.937500 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:00.937407 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 02:17:01.341204 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:01.341166 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-f89697/test-model-f89697-kserve-676c47d5c-ntkcd" podUID="1244d2b9-0346-460d-87bb-398ea191ee7b" containerName="main" containerID="cri-o://2f07226e9586eaa15fc89858df37cfc00f96a61e2b13f97838d1d3ed95228b66" gracePeriod=45 Oct 30 02:17:01.657301 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:01.657260 6221 generic.go:356] "Generic (PLEG): container finished" podID="1244d2b9-0346-460d-87bb-398ea191ee7b" containerID="2f07226e9586eaa15fc89858df37cfc00f96a61e2b13f97838d1d3ed95228b66" exitCode=0 Oct 30 02:17:01.657503 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:01.657453 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f89697/test-model-f89697-kserve-676c47d5c-ntkcd" event={"ID":"1244d2b9-0346-460d-87bb-398ea191ee7b","Type":"ContainerDied","Data":"2f07226e9586eaa15fc89858df37cfc00f96a61e2b13f97838d1d3ed95228b66"} Oct 30 02:17:01.657586 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:01.657499 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f89697/test-model-f89697-kserve-676c47d5c-ntkcd" event={"ID":"1244d2b9-0346-460d-87bb-398ea191ee7b","Type":"ContainerDied","Data":"652a2bdd63da07b012b3522cef771b94a21a9cb5e06dc6364e63a2d3361cf75f"} Oct 30 02:17:01.657586 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:01.657517 6221 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="652a2bdd63da07b012b3522cef771b94a21a9cb5e06dc6364e63a2d3361cf75f" Oct 30 02:17:01.677447 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:01.673917 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f89697/test-model-f89697-kserve-676c47d5c-ntkcd" Oct 30 02:17:01.731558 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:01.729355 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1244d2b9-0346-460d-87bb-398ea191ee7b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1244d2b9-0346-460d-87bb-398ea191ee7b-model-cache\") pod \"1244d2b9-0346-460d-87bb-398ea191ee7b\" (UID: \"1244d2b9-0346-460d-87bb-398ea191ee7b\") " Oct 30 02:17:01.731558 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:01.729487 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1244d2b9-0346-460d-87bb-398ea191ee7b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1244d2b9-0346-460d-87bb-398ea191ee7b-tmp-dir\") pod \"1244d2b9-0346-460d-87bb-398ea191ee7b\" (UID: \"1244d2b9-0346-460d-87bb-398ea191ee7b\") " Oct 30 02:17:01.731558 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:01.729585 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1244d2b9-0346-460d-87bb-398ea191ee7b-tls-certs\" (UniqueName: \"kubernetes.io/secret/1244d2b9-0346-460d-87bb-398ea191ee7b-tls-certs\") pod \"1244d2b9-0346-460d-87bb-398ea191ee7b\" (UID: \"1244d2b9-0346-460d-87bb-398ea191ee7b\") " Oct 30 02:17:01.731558 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:01.729670 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1244d2b9-0346-460d-87bb-398ea191ee7b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1244d2b9-0346-460d-87bb-398ea191ee7b-dshm\") pod \"1244d2b9-0346-460d-87bb-398ea191ee7b\" (UID: \"1244d2b9-0346-460d-87bb-398ea191ee7b\") " Oct 30 02:17:01.731558 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:01.729723 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1244d2b9-0346-460d-87bb-398ea191ee7b-home\" (UniqueName: \"kubernetes.io/empty-dir/1244d2b9-0346-460d-87bb-398ea191ee7b-home\") pod \"1244d2b9-0346-460d-87bb-398ea191ee7b\" (UID: \"1244d2b9-0346-460d-87bb-398ea191ee7b\") " Oct 30 02:17:01.731558 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:01.729777 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1244d2b9-0346-460d-87bb-398ea191ee7b-kube-api-access-smplv\" (UniqueName: \"kubernetes.io/projected/1244d2b9-0346-460d-87bb-398ea191ee7b-kube-api-access-smplv\") pod \"1244d2b9-0346-460d-87bb-398ea191ee7b\" (UID: \"1244d2b9-0346-460d-87bb-398ea191ee7b\") " Oct 30 02:17:01.776558 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:01.754813 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1244d2b9-0346-460d-87bb-398ea191ee7b-home" pod "1244d2b9-0346-460d-87bb-398ea191ee7b" (UID: "1244d2b9-0346-460d-87bb-398ea191ee7b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:01.776558 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:01.759966 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1244d2b9-0346-460d-87bb-398ea191ee7b-tmp-dir" pod "1244d2b9-0346-460d-87bb-398ea191ee7b" (UID: "1244d2b9-0346-460d-87bb-398ea191ee7b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:01.776558 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:01.761647 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1244d2b9-0346-460d-87bb-398ea191ee7b-model-cache" pod "1244d2b9-0346-460d-87bb-398ea191ee7b" (UID: "1244d2b9-0346-460d-87bb-398ea191ee7b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:01.777810 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:01.777751 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1244d2b9-0346-460d-87bb-398ea191ee7b-dshm" pod "1244d2b9-0346-460d-87bb-398ea191ee7b" (UID: "1244d2b9-0346-460d-87bb-398ea191ee7b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:01.784603 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:01.783698 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1244d2b9-0346-460d-87bb-398ea191ee7b-kube-api-access-smplv" pod "1244d2b9-0346-460d-87bb-398ea191ee7b" (UID: "1244d2b9-0346-460d-87bb-398ea191ee7b"). InnerVolumeSpecName "kube-api-access-smplv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:01.831595 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:01.830082 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1244d2b9-0346-460d-87bb-398ea191ee7b-tls-certs" pod "1244d2b9-0346-460d-87bb-398ea191ee7b" (UID: "1244d2b9-0346-460d-87bb-398ea191ee7b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:17:01.832209 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:01.832162 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smplv\" (UniqueName: \"kubernetes.io/projected/1244d2b9-0346-460d-87bb-398ea191ee7b-kube-api-access-smplv\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:17:01.832209 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:01.832195 6221 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1244d2b9-0346-460d-87bb-398ea191ee7b-model-cache\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:17:01.832209 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:01.832211 6221 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1244d2b9-0346-460d-87bb-398ea191ee7b-tmp-dir\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:17:01.832415 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:01.832223 6221 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1244d2b9-0346-460d-87bb-398ea191ee7b-tls-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:17:01.832415 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:01.832234 6221 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1244d2b9-0346-460d-87bb-398ea191ee7b-dshm\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:17:01.832415 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:01.832244 6221 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1244d2b9-0346-460d-87bb-398ea191ee7b-home\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:17:02.663734 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:02.663645 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f89697/test-model-f89697-kserve-676c47d5c-ntkcd" Oct 30 02:17:02.717421 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:02.717377 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f89697/test-model-f89697-kserve-676c47d5c-ntkcd"] Oct 30 02:17:02.729391 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:02.729357 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-f89697/test-model-f89697-kserve-676c47d5c-ntkcd"] Oct 30 02:17:04.343306 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:04.343258 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-264bdd-87b6c4ddc-ljngg"] Oct 30 02:17:04.589608 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:04.589569 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1244d2b9-0346-460d-87bb-398ea191ee7b" path="/var/lib/kubelet/pods/1244d2b9-0346-460d-87bb-398ea191ee7b/volumes" Oct 30 02:17:05.018862 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:05.018828 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-22e99d-7bc78986fb-wf66l"] Oct 30 02:17:09.343650 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:09.343606 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-264bdd-87b6c4ddc-ljngg" podUID="e9566007-9280-43dc-8c1d-0439ccc4f679" containerName="payload-pre-processing" containerID="cri-o://ebb9f5fb42508a2ba90a6a966857fb802d567d5e94dc5c35e43fac2d4e563189" gracePeriod=25 Oct 30 02:17:09.689616 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:09.689585 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-264bdd-87b6c4ddc-ljngg" Oct 30 02:17:09.765602 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:09.765526 6221 generic.go:356] "Generic (PLEG): container finished" podID="e9566007-9280-43dc-8c1d-0439ccc4f679" containerID="ebb9f5fb42508a2ba90a6a966857fb802d567d5e94dc5c35e43fac2d4e563189" exitCode=0 Oct 30 02:17:09.765817 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:09.765612 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-264bdd-87b6c4ddc-ljngg" event={"ID":"e9566007-9280-43dc-8c1d-0439ccc4f679","Type":"ContainerDied","Data":"ebb9f5fb42508a2ba90a6a966857fb802d567d5e94dc5c35e43fac2d4e563189"} Oct 30 02:17:09.765817 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:09.765668 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-264bdd-87b6c4ddc-ljngg" event={"ID":"e9566007-9280-43dc-8c1d-0439ccc4f679","Type":"ContainerDied","Data":"4e83735212562bceaa84e18aefd00e017ccc8239b7144add9908df0e274dfc95"} Oct 30 02:17:09.765817 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:09.765699 6221 scope.go:122] "RemoveContainer" containerID="ebb9f5fb42508a2ba90a6a966857fb802d567d5e94dc5c35e43fac2d4e563189" Oct 30 02:17:09.765993 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:09.765854 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-264bdd-87b6c4ddc-ljngg" Oct 30 02:17:09.778038 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:09.778007 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9566007-9280-43dc-8c1d-0439ccc4f679-kube-api-access-xjj72\" (UniqueName: \"kubernetes.io/projected/e9566007-9280-43dc-8c1d-0439ccc4f679-kube-api-access-xjj72\") pod \"e9566007-9280-43dc-8c1d-0439ccc4f679\" (UID: \"e9566007-9280-43dc-8c1d-0439ccc4f679\") " Oct 30 02:17:09.778207 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:09.778194 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9566007-9280-43dc-8c1d-0439ccc4f679-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9566007-9280-43dc-8c1d-0439ccc4f679-plugins-config-volume\") pod \"e9566007-9280-43dc-8c1d-0439ccc4f679\" (UID: \"e9566007-9280-43dc-8c1d-0439ccc4f679\") " Oct 30 02:17:09.778679 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:09.778644 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9566007-9280-43dc-8c1d-0439ccc4f679-plugins-config-volume" pod "e9566007-9280-43dc-8c1d-0439ccc4f679" (UID: "e9566007-9280-43dc-8c1d-0439ccc4f679"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:17:09.778909 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:09.778891 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9566007-9280-43dc-8c1d-0439ccc4f679-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:17:09.779235 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:09.779214 6221 scope.go:122] "RemoveContainer" containerID="ebb9f5fb42508a2ba90a6a966857fb802d567d5e94dc5c35e43fac2d4e563189" Oct 30 02:17:09.779611 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:17:09.779573 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebb9f5fb42508a2ba90a6a966857fb802d567d5e94dc5c35e43fac2d4e563189\": container with ID starting with ebb9f5fb42508a2ba90a6a966857fb802d567d5e94dc5c35e43fac2d4e563189 not found: ID does not exist" containerID="ebb9f5fb42508a2ba90a6a966857fb802d567d5e94dc5c35e43fac2d4e563189" Oct 30 02:17:09.779713 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:09.779619 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebb9f5fb42508a2ba90a6a966857fb802d567d5e94dc5c35e43fac2d4e563189"} err="failed to get container status \"ebb9f5fb42508a2ba90a6a966857fb802d567d5e94dc5c35e43fac2d4e563189\": rpc error: code = NotFound desc = could not find container \"ebb9f5fb42508a2ba90a6a966857fb802d567d5e94dc5c35e43fac2d4e563189\": container with ID starting with ebb9f5fb42508a2ba90a6a966857fb802d567d5e94dc5c35e43fac2d4e563189 not found: ID does not exist" Oct 30 02:17:09.780804 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:09.780776 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9566007-9280-43dc-8c1d-0439ccc4f679-kube-api-access-xjj72" pod "e9566007-9280-43dc-8c1d-0439ccc4f679" (UID: "e9566007-9280-43dc-8c1d-0439ccc4f679"). InnerVolumeSpecName "kube-api-access-xjj72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:09.901559 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:09.901059 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjj72\" (UniqueName: \"kubernetes.io/projected/e9566007-9280-43dc-8c1d-0439ccc4f679-kube-api-access-xjj72\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:17:10.019808 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:10.019724 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-22e99d-7bc78986fb-wf66l" podUID="0c9898f9-d745-4345-97aa-7e82f95095c6" containerName="payload-processing" containerID="cri-o://bdf8414db7276a0ee7958fff5e269580f44f229c37b2357ff6396eecd126b91e" gracePeriod=25 Oct 30 02:17:10.144155 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:10.144124 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-264bdd-87b6c4ddc-ljngg"] Oct 30 02:17:10.156241 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:10.156202 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-264bdd-87b6c4ddc-ljngg"] Oct 30 02:17:10.269282 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:10.269256 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-22e99d-7bc78986fb-wf66l" Oct 30 02:17:10.408985 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:10.408954 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c9898f9-d745-4345-97aa-7e82f95095c6-kube-api-access-sj25w\" (UniqueName: \"kubernetes.io/projected/0c9898f9-d745-4345-97aa-7e82f95095c6-kube-api-access-sj25w\") pod \"0c9898f9-d745-4345-97aa-7e82f95095c6\" (UID: \"0c9898f9-d745-4345-97aa-7e82f95095c6\") " Oct 30 02:17:10.409486 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:10.409034 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c9898f9-d745-4345-97aa-7e82f95095c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c9898f9-d745-4345-97aa-7e82f95095c6-plugins-config-volume\") pod \"0c9898f9-d745-4345-97aa-7e82f95095c6\" (UID: \"0c9898f9-d745-4345-97aa-7e82f95095c6\") " Oct 30 02:17:10.409486 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:10.409381 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c9898f9-d745-4345-97aa-7e82f95095c6-plugins-config-volume" pod "0c9898f9-d745-4345-97aa-7e82f95095c6" (UID: "0c9898f9-d745-4345-97aa-7e82f95095c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:17:10.409641 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:10.409523 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c9898f9-d745-4345-97aa-7e82f95095c6-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:17:10.411143 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:10.411121 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c9898f9-d745-4345-97aa-7e82f95095c6-kube-api-access-sj25w" pod "0c9898f9-d745-4345-97aa-7e82f95095c6" (UID: "0c9898f9-d745-4345-97aa-7e82f95095c6"). InnerVolumeSpecName "kube-api-access-sj25w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:10.510907 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:10.510874 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj25w\" (UniqueName: \"kubernetes.io/projected/0c9898f9-d745-4345-97aa-7e82f95095c6-kube-api-access-sj25w\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:17:10.548587 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:10.548553 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9566007-9280-43dc-8c1d-0439ccc4f679" path="/var/lib/kubelet/pods/e9566007-9280-43dc-8c1d-0439ccc4f679/volumes" Oct 30 02:17:10.653978 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:10.653878 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-264bdd-87b6c4ddc-ljngg" podUID="e9566007-9280-43dc-8c1d-0439ccc4f679" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.153:50052\" within 1s: context deadline exceeded" Oct 30 02:17:10.653978 ip-10-0-135-65 kubenswrapper[6221]: W1030 02:17:10.653948 6221 logging.go:55] [core] [Channel #16237 SubChannel #16238]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.153:50052", ServerName: "10.134.0.153:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.153:50052: operation was canceled" Oct 30 02:17:10.774469 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:10.774430 6221 generic.go:356] "Generic (PLEG): container finished" podID="0c9898f9-d745-4345-97aa-7e82f95095c6" containerID="bdf8414db7276a0ee7958fff5e269580f44f229c37b2357ff6396eecd126b91e" exitCode=0 Oct 30 02:17:10.774688 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:10.774475 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-22e99d-7bc78986fb-wf66l" Oct 30 02:17:10.774688 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:10.774513 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-22e99d-7bc78986fb-wf66l" event={"ID":"0c9898f9-d745-4345-97aa-7e82f95095c6","Type":"ContainerDied","Data":"bdf8414db7276a0ee7958fff5e269580f44f229c37b2357ff6396eecd126b91e"} Oct 30 02:17:10.774688 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:10.774582 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-22e99d-7bc78986fb-wf66l" event={"ID":"0c9898f9-d745-4345-97aa-7e82f95095c6","Type":"ContainerDied","Data":"5bd50cd1cd050491998c584b7c425d1130cfeeb456869335ac4abf7e4f3d732c"} Oct 30 02:17:10.774688 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:10.774609 6221 scope.go:122] "RemoveContainer" containerID="bdf8414db7276a0ee7958fff5e269580f44f229c37b2357ff6396eecd126b91e" Oct 30 02:17:10.784891 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:10.784872 6221 scope.go:122] "RemoveContainer" containerID="bdf8414db7276a0ee7958fff5e269580f44f229c37b2357ff6396eecd126b91e" Oct 30 02:17:10.785183 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:17:10.785163 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdf8414db7276a0ee7958fff5e269580f44f229c37b2357ff6396eecd126b91e\": container with ID starting with bdf8414db7276a0ee7958fff5e269580f44f229c37b2357ff6396eecd126b91e not found: ID does not exist" containerID="bdf8414db7276a0ee7958fff5e269580f44f229c37b2357ff6396eecd126b91e" Oct 30 02:17:10.785256 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:10.785194 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdf8414db7276a0ee7958fff5e269580f44f229c37b2357ff6396eecd126b91e"} err="failed to get container status \"bdf8414db7276a0ee7958fff5e269580f44f229c37b2357ff6396eecd126b91e\": rpc error: code = NotFound desc = could not find container \"bdf8414db7276a0ee7958fff5e269580f44f229c37b2357ff6396eecd126b91e\": container with ID starting with bdf8414db7276a0ee7958fff5e269580f44f229c37b2357ff6396eecd126b91e not found: ID does not exist" Oct 30 02:17:10.829394 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:10.829362 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-22e99d-7bc78986fb-wf66l"] Oct 30 02:17:10.841412 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:10.841381 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-22e99d-7bc78986fb-wf66l"] Oct 30 02:17:12.241962 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:12.241920 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f89697-d7d8df5bb-2vnvq"] Oct 30 02:17:12.548969 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:12.548935 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c9898f9-d745-4345-97aa-7e82f95095c6" path="/var/lib/kubelet/pods/0c9898f9-d745-4345-97aa-7e82f95095c6/volumes" Oct 30 02:17:14.646334 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:14.646288 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6"] Oct 30 02:17:14.647002 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:14.646566 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" podUID="6b685aa0-0c58-4911-8f31-3e920b8151a1" containerName="istio-proxy" containerID="cri-o://c164259cc14b3d0b048af1f8ee4f309f936ad612947e117f589ff7717b8d1c6b" gracePeriod=30 Oct 30 02:17:17.242411 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:17.242366 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-f89697-d7d8df5bb-2vnvq" podUID="1153eb7c-0a5d-4bae-8894-1d1519393611" containerName="payload-processing" containerID="cri-o://073c3638974349ba741349a06b535863a4427e11e176c5e3e6c076c3cec34f28" gracePeriod=25 Oct 30 02:17:17.520133 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:17.520107 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f89697-d7d8df5bb-2vnvq" Oct 30 02:17:17.635695 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:17.635651 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1153eb7c-0a5d-4bae-8894-1d1519393611-kube-api-access-zrt4f\" (UniqueName: \"kubernetes.io/projected/1153eb7c-0a5d-4bae-8894-1d1519393611-kube-api-access-zrt4f\") pod \"1153eb7c-0a5d-4bae-8894-1d1519393611\" (UID: \"1153eb7c-0a5d-4bae-8894-1d1519393611\") " Oct 30 02:17:17.635916 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:17.635756 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1153eb7c-0a5d-4bae-8894-1d1519393611-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1153eb7c-0a5d-4bae-8894-1d1519393611-plugins-config-volume\") pod \"1153eb7c-0a5d-4bae-8894-1d1519393611\" (UID: \"1153eb7c-0a5d-4bae-8894-1d1519393611\") " Oct 30 02:17:17.636194 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:17.636165 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1153eb7c-0a5d-4bae-8894-1d1519393611-plugins-config-volume" pod "1153eb7c-0a5d-4bae-8894-1d1519393611" (UID: "1153eb7c-0a5d-4bae-8894-1d1519393611"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:17:17.636418 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:17.636402 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1153eb7c-0a5d-4bae-8894-1d1519393611-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:17:17.637903 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:17.637881 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1153eb7c-0a5d-4bae-8894-1d1519393611-kube-api-access-zrt4f" pod "1153eb7c-0a5d-4bae-8894-1d1519393611" (UID: "1153eb7c-0a5d-4bae-8894-1d1519393611"). InnerVolumeSpecName "kube-api-access-zrt4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:17.737972 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:17.737941 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrt4f\" (UniqueName: \"kubernetes.io/projected/1153eb7c-0a5d-4bae-8894-1d1519393611-kube-api-access-zrt4f\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:17:17.854014 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:17.853977 6221 generic.go:356] "Generic (PLEG): container finished" podID="1153eb7c-0a5d-4bae-8894-1d1519393611" containerID="073c3638974349ba741349a06b535863a4427e11e176c5e3e6c076c3cec34f28" exitCode=0 Oct 30 02:17:17.854235 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:17.854027 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f89697-d7d8df5bb-2vnvq" Oct 30 02:17:17.854235 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:17.854060 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f89697-d7d8df5bb-2vnvq" event={"ID":"1153eb7c-0a5d-4bae-8894-1d1519393611","Type":"ContainerDied","Data":"073c3638974349ba741349a06b535863a4427e11e176c5e3e6c076c3cec34f28"} Oct 30 02:17:17.854235 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:17.854126 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f89697-d7d8df5bb-2vnvq" event={"ID":"1153eb7c-0a5d-4bae-8894-1d1519393611","Type":"ContainerDied","Data":"2bf99f14f401bf803c2aadbc7c794e62756bd3d621dcc432ae55314a45e30ceb"} Oct 30 02:17:17.854235 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:17.854158 6221 scope.go:122] "RemoveContainer" containerID="073c3638974349ba741349a06b535863a4427e11e176c5e3e6c076c3cec34f28" Oct 30 02:17:17.865247 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:17.865229 6221 scope.go:122] "RemoveContainer" containerID="073c3638974349ba741349a06b535863a4427e11e176c5e3e6c076c3cec34f28" Oct 30 02:17:17.865554 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:17:17.865519 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"073c3638974349ba741349a06b535863a4427e11e176c5e3e6c076c3cec34f28\": container with ID starting with 073c3638974349ba741349a06b535863a4427e11e176c5e3e6c076c3cec34f28 not found: ID does not exist" containerID="073c3638974349ba741349a06b535863a4427e11e176c5e3e6c076c3cec34f28" Oct 30 02:17:17.865618 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:17.865566 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"073c3638974349ba741349a06b535863a4427e11e176c5e3e6c076c3cec34f28"} err="failed to get container status \"073c3638974349ba741349a06b535863a4427e11e176c5e3e6c076c3cec34f28\": rpc error: code = NotFound desc = could not find container \"073c3638974349ba741349a06b535863a4427e11e176c5e3e6c076c3cec34f28\": container with ID starting with 073c3638974349ba741349a06b535863a4427e11e176c5e3e6c076c3cec34f28 not found: ID does not exist" Oct 30 02:17:17.910312 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:17.910278 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f89697-d7d8df5bb-2vnvq"] Oct 30 02:17:17.920393 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:17.920369 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f89697-d7d8df5bb-2vnvq"] Oct 30 02:17:18.349522 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:18.349482 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-803472-866c95bf4c-jfvc2"] Oct 30 02:17:18.350018 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:18.349713 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-803472-866c95bf4c-jfvc2" podUID="7562db53-8060-4df5-a62d-699d3a44caa9" containerName="maas-api" containerID="cri-o://495f4da85da3f5a99b78c70f6b35baab54766ac332d0cfb31d9008fee19f93f5" gracePeriod=30 Oct 30 02:17:18.548696 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:18.548660 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1153eb7c-0a5d-4bae-8894-1d1519393611" path="/var/lib/kubelet/pods/1153eb7c-0a5d-4bae-8894-1d1519393611/volumes" Oct 30 02:17:18.698038 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:18.698008 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-803472-866c95bf4c-jfvc2" Oct 30 02:17:18.881325 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:18.881111 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7562db53-8060-4df5-a62d-699d3a44caa9-kube-api-access-29zpc\" (UniqueName: \"kubernetes.io/projected/7562db53-8060-4df5-a62d-699d3a44caa9-kube-api-access-29zpc\") pod \"7562db53-8060-4df5-a62d-699d3a44caa9\" (UID: \"7562db53-8060-4df5-a62d-699d3a44caa9\") " Oct 30 02:17:18.881325 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:18.881200 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7562db53-8060-4df5-a62d-699d3a44caa9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7562db53-8060-4df5-a62d-699d3a44caa9-maas-api-tls\") pod \"7562db53-8060-4df5-a62d-699d3a44caa9\" (UID: \"7562db53-8060-4df5-a62d-699d3a44caa9\") " Oct 30 02:17:18.881325 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:18.881279 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7562db53-8060-4df5-a62d-699d3a44caa9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7562db53-8060-4df5-a62d-699d3a44caa9-metrics-certs\") pod \"7562db53-8060-4df5-a62d-699d3a44caa9\" (UID: \"7562db53-8060-4df5-a62d-699d3a44caa9\") " Oct 30 02:17:18.881325 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:18.881315 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7562db53-8060-4df5-a62d-699d3a44caa9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7562db53-8060-4df5-a62d-699d3a44caa9-openshift-service-ca\") pod \"7562db53-8060-4df5-a62d-699d3a44caa9\" (UID: \"7562db53-8060-4df5-a62d-699d3a44caa9\") " Oct 30 02:17:18.882742 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:18.882440 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7562db53-8060-4df5-a62d-699d3a44caa9-openshift-service-ca" pod "7562db53-8060-4df5-a62d-699d3a44caa9" (UID: "7562db53-8060-4df5-a62d-699d3a44caa9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:17:18.883582 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:18.883559 6221 generic.go:356] "Generic (PLEG): container finished" podID="7562db53-8060-4df5-a62d-699d3a44caa9" containerID="495f4da85da3f5a99b78c70f6b35baab54766ac332d0cfb31d9008fee19f93f5" exitCode=0 Oct 30 02:17:18.883806 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:18.883606 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-803472-866c95bf4c-jfvc2" event={"ID":"7562db53-8060-4df5-a62d-699d3a44caa9","Type":"ContainerDied","Data":"495f4da85da3f5a99b78c70f6b35baab54766ac332d0cfb31d9008fee19f93f5"} Oct 30 02:17:18.883891 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:18.883861 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-803472-866c95bf4c-jfvc2" event={"ID":"7562db53-8060-4df5-a62d-699d3a44caa9","Type":"ContainerDied","Data":"74fb9d209a20339c844f9383d5957e1d341c6a9607bcceab2b2bd9367089f45b"} Oct 30 02:17:18.883962 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:18.883895 6221 scope.go:122] "RemoveContainer" containerID="495f4da85da3f5a99b78c70f6b35baab54766ac332d0cfb31d9008fee19f93f5" Oct 30 02:17:18.884036 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:18.883634 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-803472-866c95bf4c-jfvc2" Oct 30 02:17:18.891249 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:18.891222 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7562db53-8060-4df5-a62d-699d3a44caa9-kube-api-access-29zpc" pod "7562db53-8060-4df5-a62d-699d3a44caa9" (UID: "7562db53-8060-4df5-a62d-699d3a44caa9"). InnerVolumeSpecName "kube-api-access-29zpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:18.896826 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:18.896791 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7562db53-8060-4df5-a62d-699d3a44caa9-maas-api-tls" pod "7562db53-8060-4df5-a62d-699d3a44caa9" (UID: "7562db53-8060-4df5-a62d-699d3a44caa9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:17:18.899890 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:18.899868 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7562db53-8060-4df5-a62d-699d3a44caa9-metrics-certs" pod "7562db53-8060-4df5-a62d-699d3a44caa9" (UID: "7562db53-8060-4df5-a62d-699d3a44caa9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:17:18.983701 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:18.983665 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29zpc\" (UniqueName: \"kubernetes.io/projected/7562db53-8060-4df5-a62d-699d3a44caa9-kube-api-access-29zpc\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:17:18.983701 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:18.983692 6221 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7562db53-8060-4df5-a62d-699d3a44caa9-maas-api-tls\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:17:18.983701 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:18.983703 6221 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7562db53-8060-4df5-a62d-699d3a44caa9-metrics-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:17:18.983701 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:18.983712 6221 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7562db53-8060-4df5-a62d-699d3a44caa9-openshift-service-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:17:19.012028 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:19.012005 6221 scope.go:122] "RemoveContainer" containerID="495f4da85da3f5a99b78c70f6b35baab54766ac332d0cfb31d9008fee19f93f5" Oct 30 02:17:19.012338 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:17:19.012311 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"495f4da85da3f5a99b78c70f6b35baab54766ac332d0cfb31d9008fee19f93f5\": container with ID starting with 495f4da85da3f5a99b78c70f6b35baab54766ac332d0cfb31d9008fee19f93f5 not found: ID does not exist" containerID="495f4da85da3f5a99b78c70f6b35baab54766ac332d0cfb31d9008fee19f93f5" Oct 30 02:17:19.012388 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:19.012346 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"495f4da85da3f5a99b78c70f6b35baab54766ac332d0cfb31d9008fee19f93f5"} err="failed to get container status \"495f4da85da3f5a99b78c70f6b35baab54766ac332d0cfb31d9008fee19f93f5\": rpc error: code = NotFound desc = could not find container \"495f4da85da3f5a99b78c70f6b35baab54766ac332d0cfb31d9008fee19f93f5\": container with ID starting with 495f4da85da3f5a99b78c70f6b35baab54766ac332d0cfb31d9008fee19f93f5 not found: ID does not exist" Oct 30 02:17:19.243588 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:19.243517 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-803472-866c95bf4c-jfvc2"] Oct 30 02:17:19.247260 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:19.247229 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-803472-866c95bf4c-jfvc2"] Oct 30 02:17:19.894428 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:19.894393 6221 generic.go:356] "Generic (PLEG): container finished" podID="6b685aa0-0c58-4911-8f31-3e920b8151a1" containerID="c164259cc14b3d0b048af1f8ee4f309f936ad612947e117f589ff7717b8d1c6b" exitCode=0 Oct 30 02:17:19.894929 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:19.894488 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" event={"ID":"6b685aa0-0c58-4911-8f31-3e920b8151a1","Type":"ContainerDied","Data":"c164259cc14b3d0b048af1f8ee4f309f936ad612947e117f589ff7717b8d1c6b"} Oct 30 02:17:19.894929 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:19.894598 6221 scope.go:122] "RemoveContainer" containerID="10902634aa9bb9a9f3db67ba6c2317fe28fee31b321eb17ddfaa4f4be3a78eb4" Oct 30 02:17:19.927369 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:19.927344 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" Oct 30 02:17:19.995616 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:19.995580 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b685aa0-0c58-4911-8f31-3e920b8151a1-istio-token\" (UniqueName: \"kubernetes.io/projected/6b685aa0-0c58-4911-8f31-3e920b8151a1-istio-token\") pod \"6b685aa0-0c58-4911-8f31-3e920b8151a1\" (UID: \"6b685aa0-0c58-4911-8f31-3e920b8151a1\") " Oct 30 02:17:19.995818 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:19.995630 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b685aa0-0c58-4911-8f31-3e920b8151a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b685aa0-0c58-4911-8f31-3e920b8151a1-workload-socket\") pod \"6b685aa0-0c58-4911-8f31-3e920b8151a1\" (UID: \"6b685aa0-0c58-4911-8f31-3e920b8151a1\") " Oct 30 02:17:19.995818 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:19.995668 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b685aa0-0c58-4911-8f31-3e920b8151a1-kube-api-access-chljc\" (UniqueName: \"kubernetes.io/projected/6b685aa0-0c58-4911-8f31-3e920b8151a1-kube-api-access-chljc\") pod \"6b685aa0-0c58-4911-8f31-3e920b8151a1\" (UID: \"6b685aa0-0c58-4911-8f31-3e920b8151a1\") " Oct 30 02:17:19.995818 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:19.995776 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b685aa0-0c58-4911-8f31-3e920b8151a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b685aa0-0c58-4911-8f31-3e920b8151a1-credential-socket\") pod \"6b685aa0-0c58-4911-8f31-3e920b8151a1\" (UID: \"6b685aa0-0c58-4911-8f31-3e920b8151a1\") " Oct 30 02:17:19.995818 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:19.995810 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b685aa0-0c58-4911-8f31-3e920b8151a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b685aa0-0c58-4911-8f31-3e920b8151a1-workload-certs\") pod \"6b685aa0-0c58-4911-8f31-3e920b8151a1\" (UID: \"6b685aa0-0c58-4911-8f31-3e920b8151a1\") " Oct 30 02:17:19.996094 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:19.995840 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b685aa0-0c58-4911-8f31-3e920b8151a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b685aa0-0c58-4911-8f31-3e920b8151a1-istiod-ca-cert\") pod \"6b685aa0-0c58-4911-8f31-3e920b8151a1\" (UID: \"6b685aa0-0c58-4911-8f31-3e920b8151a1\") " Oct 30 02:17:19.996094 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:19.995893 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b685aa0-0c58-4911-8f31-3e920b8151a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b685aa0-0c58-4911-8f31-3e920b8151a1-istio-data\") pod \"6b685aa0-0c58-4911-8f31-3e920b8151a1\" (UID: \"6b685aa0-0c58-4911-8f31-3e920b8151a1\") " Oct 30 02:17:19.996094 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:19.995991 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b685aa0-0c58-4911-8f31-3e920b8151a1-workload-socket" pod "6b685aa0-0c58-4911-8f31-3e920b8151a1" (UID: "6b685aa0-0c58-4911-8f31-3e920b8151a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:19.996094 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:19.996009 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b685aa0-0c58-4911-8f31-3e920b8151a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b685aa0-0c58-4911-8f31-3e920b8151a1-istio-podinfo\") pod \"6b685aa0-0c58-4911-8f31-3e920b8151a1\" (UID: \"6b685aa0-0c58-4911-8f31-3e920b8151a1\") " Oct 30 02:17:19.996295 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:19.996111 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b685aa0-0c58-4911-8f31-3e920b8151a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b685aa0-0c58-4911-8f31-3e920b8151a1-istio-envoy\") pod \"6b685aa0-0c58-4911-8f31-3e920b8151a1\" (UID: \"6b685aa0-0c58-4911-8f31-3e920b8151a1\") " Oct 30 02:17:19.996295 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:19.996198 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b685aa0-0c58-4911-8f31-3e920b8151a1-workload-certs" pod "6b685aa0-0c58-4911-8f31-3e920b8151a1" (UID: "6b685aa0-0c58-4911-8f31-3e920b8151a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:19.996405 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:19.996330 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b685aa0-0c58-4911-8f31-3e920b8151a1-istiod-ca-cert" pod "6b685aa0-0c58-4911-8f31-3e920b8151a1" (UID: "6b685aa0-0c58-4911-8f31-3e920b8151a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:17:19.996405 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:19.996372 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b685aa0-0c58-4911-8f31-3e920b8151a1-credential-socket" pod "6b685aa0-0c58-4911-8f31-3e920b8151a1" (UID: "6b685aa0-0c58-4911-8f31-3e920b8151a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:19.996806 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:19.996694 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b685aa0-0c58-4911-8f31-3e920b8151a1-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:17:19.996806 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:19.996723 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b685aa0-0c58-4911-8f31-3e920b8151a1-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:17:19.996806 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:19.996738 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b685aa0-0c58-4911-8f31-3e920b8151a1-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:17:19.996806 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:19.996750 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b685aa0-0c58-4911-8f31-3e920b8151a1-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:17:19.998445 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:19.998418 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b685aa0-0c58-4911-8f31-3e920b8151a1-kube-api-access-chljc" pod "6b685aa0-0c58-4911-8f31-3e920b8151a1" (UID: "6b685aa0-0c58-4911-8f31-3e920b8151a1"). InnerVolumeSpecName "kube-api-access-chljc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:19.998711 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:19.998690 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b685aa0-0c58-4911-8f31-3e920b8151a1-istio-envoy" pod "6b685aa0-0c58-4911-8f31-3e920b8151a1" (UID: "6b685aa0-0c58-4911-8f31-3e920b8151a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:19.999985 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:19.999956 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b685aa0-0c58-4911-8f31-3e920b8151a1-istio-data" pod "6b685aa0-0c58-4911-8f31-3e920b8151a1" (UID: "6b685aa0-0c58-4911-8f31-3e920b8151a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:20.000992 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:20.000969 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b685aa0-0c58-4911-8f31-3e920b8151a1-istio-podinfo" pod "6b685aa0-0c58-4911-8f31-3e920b8151a1" (UID: "6b685aa0-0c58-4911-8f31-3e920b8151a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:17:20.001137 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:20.001006 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b685aa0-0c58-4911-8f31-3e920b8151a1-istio-token" pod "6b685aa0-0c58-4911-8f31-3e920b8151a1" (UID: "6b685aa0-0c58-4911-8f31-3e920b8151a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:20.098264 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:20.098228 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b685aa0-0c58-4911-8f31-3e920b8151a1-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:17:20.098264 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:20.098259 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b685aa0-0c58-4911-8f31-3e920b8151a1-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:17:20.098264 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:20.098269 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b685aa0-0c58-4911-8f31-3e920b8151a1-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:17:20.098506 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:20.098277 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b685aa0-0c58-4911-8f31-3e920b8151a1-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:17:20.098506 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:20.098286 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chljc\" (UniqueName: \"kubernetes.io/projected/6b685aa0-0c58-4911-8f31-3e920b8151a1-kube-api-access-chljc\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:17:20.548299 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:20.548265 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7562db53-8060-4df5-a62d-699d3a44caa9" path="/var/lib/kubelet/pods/7562db53-8060-4df5-a62d-699d3a44caa9/volumes" Oct 30 02:17:20.930903 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:20.928424 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" event={"ID":"6b685aa0-0c58-4911-8f31-3e920b8151a1","Type":"ContainerDied","Data":"814dd9f5f2e7b29fcf71d358efddcd4e918a5f5e062f894134794d8026fe12eb"} Oct 30 02:17:20.930903 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:20.928500 6221 scope.go:122] "RemoveContainer" containerID="c164259cc14b3d0b048af1f8ee4f309f936ad612947e117f589ff7717b8d1c6b" Oct 30 02:17:20.930903 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:20.928677 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6" Oct 30 02:17:21.038678 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:21.038643 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6"] Oct 30 02:17:21.043044 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:21.043016 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-264bdd-openshift-default-5bdcf995c7-c4xr6"] Oct 30 02:17:21.495393 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:17:21.495354 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 02:17:22.548905 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:22.548873 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b685aa0-0c58-4911-8f31-3e920b8151a1" path="/var/lib/kubelet/pods/6b685aa0-0c58-4911-8f31-3e920b8151a1/volumes" Oct 30 02:17:24.552423 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:24.552387 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k"] Oct 30 02:17:24.552936 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:24.552692 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k" podUID="ff0dc863-4ea6-454f-9f61-5bfd9a9db849" containerName="istio-proxy" containerID="cri-o://bf266f7b670d5a0abe14a1284abce009f29ad5cce80a62ea132ee6c59e9526a4" gracePeriod=30 Oct 30 02:17:29.831115 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:29.831077 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477"] Oct 30 02:17:29.831703 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:29.831347 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" podUID="d0218485-8862-4117-902d-d5de5c8c963b" containerName="istio-proxy" containerID="cri-o://928f5cff627b8d17ccd90cfd16bd06e03a1891d2f6f6cff47babb39627a0e8dc" gracePeriod=30 Oct 30 02:17:29.850329 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:29.850307 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k" Oct 30 02:17:29.991552 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:29.983955 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-kube-api-access-mc9f5\" (UniqueName: \"kubernetes.io/projected/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-kube-api-access-mc9f5\") pod \"ff0dc863-4ea6-454f-9f61-5bfd9a9db849\" (UID: \"ff0dc863-4ea6-454f-9f61-5bfd9a9db849\") " Oct 30 02:17:29.991552 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:29.984055 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-istiod-ca-cert\") pod \"ff0dc863-4ea6-454f-9f61-5bfd9a9db849\" (UID: \"ff0dc863-4ea6-454f-9f61-5bfd9a9db849\") " Oct 30 02:17:29.991552 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:29.984137 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-workload-socket\") pod \"ff0dc863-4ea6-454f-9f61-5bfd9a9db849\" (UID: \"ff0dc863-4ea6-454f-9f61-5bfd9a9db849\") " Oct 30 02:17:29.991552 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:29.984212 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-credential-socket\") pod \"ff0dc863-4ea6-454f-9f61-5bfd9a9db849\" (UID: \"ff0dc863-4ea6-454f-9f61-5bfd9a9db849\") " Oct 30 02:17:29.991552 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:29.984303 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-istio-envoy\") pod \"ff0dc863-4ea6-454f-9f61-5bfd9a9db849\" (UID: \"ff0dc863-4ea6-454f-9f61-5bfd9a9db849\") " Oct 30 02:17:29.991552 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:29.984375 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-istio-podinfo\") pod \"ff0dc863-4ea6-454f-9f61-5bfd9a9db849\" (UID: \"ff0dc863-4ea6-454f-9f61-5bfd9a9db849\") " Oct 30 02:17:29.991552 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:29.984444 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-istio-token\" (UniqueName: \"kubernetes.io/projected/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-istio-token\") pod \"ff0dc863-4ea6-454f-9f61-5bfd9a9db849\" (UID: \"ff0dc863-4ea6-454f-9f61-5bfd9a9db849\") " Oct 30 02:17:29.991552 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:29.984479 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-workload-certs\") pod \"ff0dc863-4ea6-454f-9f61-5bfd9a9db849\" (UID: \"ff0dc863-4ea6-454f-9f61-5bfd9a9db849\") " Oct 30 02:17:29.991552 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:29.984545 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-istio-data\") pod \"ff0dc863-4ea6-454f-9f61-5bfd9a9db849\" (UID: \"ff0dc863-4ea6-454f-9f61-5bfd9a9db849\") " Oct 30 02:17:29.991552 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:29.990501 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-istio-data" pod "ff0dc863-4ea6-454f-9f61-5bfd9a9db849" (UID: "ff0dc863-4ea6-454f-9f61-5bfd9a9db849"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:29.991552 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:29.991351 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-workload-certs" pod "ff0dc863-4ea6-454f-9f61-5bfd9a9db849" (UID: "ff0dc863-4ea6-454f-9f61-5bfd9a9db849"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:30.007777 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:29.997913 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-credential-socket" pod "ff0dc863-4ea6-454f-9f61-5bfd9a9db849" (UID: "ff0dc863-4ea6-454f-9f61-5bfd9a9db849"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:30.007777 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:29.998137 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-workload-socket" pod "ff0dc863-4ea6-454f-9f61-5bfd9a9db849" (UID: "ff0dc863-4ea6-454f-9f61-5bfd9a9db849"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:30.007777 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:30.005263 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-istiod-ca-cert" pod "ff0dc863-4ea6-454f-9f61-5bfd9a9db849" (UID: "ff0dc863-4ea6-454f-9f61-5bfd9a9db849"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:17:30.036597 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:30.033364 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-istio-podinfo" pod "ff0dc863-4ea6-454f-9f61-5bfd9a9db849" (UID: "ff0dc863-4ea6-454f-9f61-5bfd9a9db849"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:17:30.037007 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:30.036931 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-kube-api-access-mc9f5" pod "ff0dc863-4ea6-454f-9f61-5bfd9a9db849" (UID: "ff0dc863-4ea6-454f-9f61-5bfd9a9db849"). InnerVolumeSpecName "kube-api-access-mc9f5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:30.038868 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:30.038820 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-istio-token" pod "ff0dc863-4ea6-454f-9f61-5bfd9a9db849" (UID: "ff0dc863-4ea6-454f-9f61-5bfd9a9db849"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:30.039408 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:30.039369 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-istio-envoy" pod "ff0dc863-4ea6-454f-9f61-5bfd9a9db849" (UID: "ff0dc863-4ea6-454f-9f61-5bfd9a9db849"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:30.043862 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:30.039811 6221 generic.go:356] "Generic (PLEG): container finished" podID="ff0dc863-4ea6-454f-9f61-5bfd9a9db849" containerID="bf266f7b670d5a0abe14a1284abce009f29ad5cce80a62ea132ee6c59e9526a4" exitCode=0 Oct 30 02:17:30.043862 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:30.040077 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k" event={"ID":"ff0dc863-4ea6-454f-9f61-5bfd9a9db849","Type":"ContainerDied","Data":"bf266f7b670d5a0abe14a1284abce009f29ad5cce80a62ea132ee6c59e9526a4"} Oct 30 02:17:30.043862 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:30.040137 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k" event={"ID":"ff0dc863-4ea6-454f-9f61-5bfd9a9db849","Type":"ContainerDied","Data":"30577fbf9c731884e7e42e83bc9072a62094e348057c5a926acc2c5311c1ce74"} Oct 30 02:17:30.043862 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:30.040170 6221 scope.go:122] "RemoveContainer" containerID="bf266f7b670d5a0abe14a1284abce009f29ad5cce80a62ea132ee6c59e9526a4" Oct 30 02:17:30.043862 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:30.040361 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k" Oct 30 02:17:30.064307 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:30.064284 6221 scope.go:122] "RemoveContainer" containerID="bf266f7b670d5a0abe14a1284abce009f29ad5cce80a62ea132ee6c59e9526a4" Oct 30 02:17:30.064745 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:17:30.064717 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf266f7b670d5a0abe14a1284abce009f29ad5cce80a62ea132ee6c59e9526a4\": container with ID starting with bf266f7b670d5a0abe14a1284abce009f29ad5cce80a62ea132ee6c59e9526a4 not found: ID does not exist" containerID="bf266f7b670d5a0abe14a1284abce009f29ad5cce80a62ea132ee6c59e9526a4" Oct 30 02:17:30.064843 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:30.064756 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf266f7b670d5a0abe14a1284abce009f29ad5cce80a62ea132ee6c59e9526a4"} err="failed to get container status \"bf266f7b670d5a0abe14a1284abce009f29ad5cce80a62ea132ee6c59e9526a4\": rpc error: code = NotFound desc = could not find container \"bf266f7b670d5a0abe14a1284abce009f29ad5cce80a62ea132ee6c59e9526a4\": container with ID starting with bf266f7b670d5a0abe14a1284abce009f29ad5cce80a62ea132ee6c59e9526a4 not found: ID does not exist" Oct 30 02:17:30.086868 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:30.086794 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:17:30.086868 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:30.086825 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:17:30.086868 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:30.086839 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:17:30.086868 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:30.086849 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:17:30.086868 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:30.086857 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:17:30.086868 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:30.086864 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:17:30.086868 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:30.086872 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:17:30.087197 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:30.086880 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc9f5\" (UniqueName: \"kubernetes.io/projected/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-kube-api-access-mc9f5\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:17:30.087197 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:30.086889 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff0dc863-4ea6-454f-9f61-5bfd9a9db849-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:17:30.097623 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:30.097589 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k"] Oct 30 02:17:30.101731 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:30.101705 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-f89697-openshift-default-7f6dfbfd45-hr97k"] Oct 30 02:17:30.110955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:30.110927 6221 patch_prober.go:29] interesting pod/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 02:17:30.111072 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:30.110992 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" podUID="d0218485-8862-4117-902d-d5de5c8c963b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 02:17:30.561261 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:30.560452 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff0dc863-4ea6-454f-9f61-5bfd9a9db849" path="/var/lib/kubelet/pods/ff0dc863-4ea6-454f-9f61-5bfd9a9db849/volumes" Oct 30 02:17:35.079011 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:35.078972 6221 generic.go:356] "Generic (PLEG): container finished" podID="d0218485-8862-4117-902d-d5de5c8c963b" containerID="928f5cff627b8d17ccd90cfd16bd06e03a1891d2f6f6cff47babb39627a0e8dc" exitCode=0 Oct 30 02:17:35.079486 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:35.079048 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" event={"ID":"d0218485-8862-4117-902d-d5de5c8c963b","Type":"ContainerDied","Data":"928f5cff627b8d17ccd90cfd16bd06e03a1891d2f6f6cff47babb39627a0e8dc"} Oct 30 02:17:35.079486 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:35.079125 6221 scope.go:122] "RemoveContainer" containerID="c6e56cdfe0cf40702b3e4f5918a6a69450df03db19b2f4dac896ae545228e79d" Oct 30 02:17:35.123007 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:35.122957 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" Oct 30 02:17:35.148462 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:35.148437 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0218485-8862-4117-902d-d5de5c8c963b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0218485-8862-4117-902d-d5de5c8c963b-istio-envoy\") pod \"d0218485-8862-4117-902d-d5de5c8c963b\" (UID: \"d0218485-8862-4117-902d-d5de5c8c963b\") " Oct 30 02:17:35.148588 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:35.148469 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0218485-8862-4117-902d-d5de5c8c963b-istio-token\" (UniqueName: \"kubernetes.io/projected/d0218485-8862-4117-902d-d5de5c8c963b-istio-token\") pod \"d0218485-8862-4117-902d-d5de5c8c963b\" (UID: \"d0218485-8862-4117-902d-d5de5c8c963b\") " Oct 30 02:17:35.148588 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:35.148492 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0218485-8862-4117-902d-d5de5c8c963b-kube-api-access-7z9vc\" (UniqueName: \"kubernetes.io/projected/d0218485-8862-4117-902d-d5de5c8c963b-kube-api-access-7z9vc\") pod \"d0218485-8862-4117-902d-d5de5c8c963b\" (UID: \"d0218485-8862-4117-902d-d5de5c8c963b\") " Oct 30 02:17:35.148677 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:35.148661 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0218485-8862-4117-902d-d5de5c8c963b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0218485-8862-4117-902d-d5de5c8c963b-credential-socket\") pod \"d0218485-8862-4117-902d-d5de5c8c963b\" (UID: \"d0218485-8862-4117-902d-d5de5c8c963b\") " Oct 30 02:17:35.148713 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:35.148694 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0218485-8862-4117-902d-d5de5c8c963b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0218485-8862-4117-902d-d5de5c8c963b-istio-data\") pod \"d0218485-8862-4117-902d-d5de5c8c963b\" (UID: \"d0218485-8862-4117-902d-d5de5c8c963b\") " Oct 30 02:17:35.148766 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:35.148736 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0218485-8862-4117-902d-d5de5c8c963b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0218485-8862-4117-902d-d5de5c8c963b-istio-podinfo\") pod \"d0218485-8862-4117-902d-d5de5c8c963b\" (UID: \"d0218485-8862-4117-902d-d5de5c8c963b\") " Oct 30 02:17:35.148848 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:35.148820 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0218485-8862-4117-902d-d5de5c8c963b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0218485-8862-4117-902d-d5de5c8c963b-workload-socket\") pod \"d0218485-8862-4117-902d-d5de5c8c963b\" (UID: \"d0218485-8862-4117-902d-d5de5c8c963b\") " Oct 30 02:17:35.148917 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:35.148907 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0218485-8862-4117-902d-d5de5c8c963b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0218485-8862-4117-902d-d5de5c8c963b-workload-certs\") pod \"d0218485-8862-4117-902d-d5de5c8c963b\" (UID: \"d0218485-8862-4117-902d-d5de5c8c963b\") " Oct 30 02:17:35.148978 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:35.148951 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0218485-8862-4117-902d-d5de5c8c963b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0218485-8862-4117-902d-d5de5c8c963b-istiod-ca-cert\") pod \"d0218485-8862-4117-902d-d5de5c8c963b\" (UID: \"d0218485-8862-4117-902d-d5de5c8c963b\") " Oct 30 02:17:35.149255 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:35.148945 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0218485-8862-4117-902d-d5de5c8c963b-credential-socket" pod "d0218485-8862-4117-902d-d5de5c8c963b" (UID: "d0218485-8862-4117-902d-d5de5c8c963b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:35.149255 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:35.149102 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0218485-8862-4117-902d-d5de5c8c963b-istio-data" pod "d0218485-8862-4117-902d-d5de5c8c963b" (UID: "d0218485-8862-4117-902d-d5de5c8c963b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:35.149255 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:35.149114 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0218485-8862-4117-902d-d5de5c8c963b-workload-socket" pod "d0218485-8862-4117-902d-d5de5c8c963b" (UID: "d0218485-8862-4117-902d-d5de5c8c963b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:35.149255 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:35.149158 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0218485-8862-4117-902d-d5de5c8c963b-workload-certs" pod "d0218485-8862-4117-902d-d5de5c8c963b" (UID: "d0218485-8862-4117-902d-d5de5c8c963b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:35.149683 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:35.149409 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0218485-8862-4117-902d-d5de5c8c963b-istiod-ca-cert" pod "d0218485-8862-4117-902d-d5de5c8c963b" (UID: "d0218485-8862-4117-902d-d5de5c8c963b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:17:35.149683 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:35.149631 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0218485-8862-4117-902d-d5de5c8c963b-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:17:35.149683 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:35.149652 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0218485-8862-4117-902d-d5de5c8c963b-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:17:35.149683 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:35.149662 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0218485-8862-4117-902d-d5de5c8c963b-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:17:35.149683 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:35.149670 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0218485-8862-4117-902d-d5de5c8c963b-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:17:35.149683 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:35.149678 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0218485-8862-4117-902d-d5de5c8c963b-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:17:35.150920 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:35.150895 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0218485-8862-4117-902d-d5de5c8c963b-istio-envoy" pod "d0218485-8862-4117-902d-d5de5c8c963b" (UID: "d0218485-8862-4117-902d-d5de5c8c963b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:17:35.151038 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:35.150983 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0218485-8862-4117-902d-d5de5c8c963b-kube-api-access-7z9vc" pod "d0218485-8862-4117-902d-d5de5c8c963b" (UID: "d0218485-8862-4117-902d-d5de5c8c963b"). InnerVolumeSpecName "kube-api-access-7z9vc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:35.151103 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:35.151067 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0218485-8862-4117-902d-d5de5c8c963b-istio-token" pod "d0218485-8862-4117-902d-d5de5c8c963b" (UID: "d0218485-8862-4117-902d-d5de5c8c963b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:17:35.151163 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:35.151135 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0218485-8862-4117-902d-d5de5c8c963b-istio-podinfo" pod "d0218485-8862-4117-902d-d5de5c8c963b" (UID: "d0218485-8862-4117-902d-d5de5c8c963b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:17:35.250863 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:35.250819 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0218485-8862-4117-902d-d5de5c8c963b-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:17:35.250863 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:35.250854 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0218485-8862-4117-902d-d5de5c8c963b-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:17:35.250863 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:35.250867 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z9vc\" (UniqueName: \"kubernetes.io/projected/d0218485-8862-4117-902d-d5de5c8c963b-kube-api-access-7z9vc\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:17:35.251167 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:35.250881 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0218485-8862-4117-902d-d5de5c8c963b-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:17:36.088179 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:36.088154 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" Oct 30 02:17:36.088179 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:36.088174 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477" event={"ID":"d0218485-8862-4117-902d-d5de5c8c963b","Type":"ContainerDied","Data":"3a53452d0c89200b67b6941bef2d245e4c72bf6fb4e8f5cdb8065cd035027091"} Oct 30 02:17:36.088696 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:36.088232 6221 scope.go:122] "RemoveContainer" containerID="928f5cff627b8d17ccd90cfd16bd06e03a1891d2f6f6cff47babb39627a0e8dc" Oct 30 02:17:36.199795 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:36.199752 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477"] Oct 30 02:17:36.209770 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:36.209739 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-22e99d-openshift-default-8bf65f6dc-bv477"] Oct 30 02:17:36.547831 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:36.547800 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0218485-8862-4117-902d-d5de5c8c963b" path="/var/lib/kubelet/pods/d0218485-8862-4117-902d-d5de5c8c963b/volumes" Oct 30 02:17:40.045837 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:40.045805 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 02:17:51.495078 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:17:51.495039 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 02:17:55.093375 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:55.093343 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv"] Oct 30 02:17:55.094294 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:55.093647 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv" podUID="1c9548ee-90ef-4fe2-9412-4da31c20dfba" containerName="istio-proxy" containerID="cri-o://d828962be39b77681f25d73fdb539631b9c68dad4027b7262ee0cff69484f110" gracePeriod=30 Oct 30 02:17:55.098133 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:55.098105 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd"] Oct 30 02:17:55.098342 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:55.098319 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd" podUID="61c55830-d642-4daa-8692-5f4f409b9a1a" containerName="istio-proxy" containerID="cri-o://517da38a773a77e45280517c2ed45aaf470a5296a9e26718e796ab3dc4858c42" gracePeriod=30 Oct 30 02:17:57.289997 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:57.289966 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2vsns_d332bbb4-c699-44e8-96ba-516d8ca03e95/ovn-acl-logging/0.log" Oct 30 02:17:57.293424 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:57.293401 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2vsns_d332bbb4-c699-44e8-96ba-516d8ca03e95/ovn-acl-logging/0.log" Oct 30 02:17:58.505971 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:17:58.505943 6221 scope.go:122] "RemoveContainer" containerID="2f07226e9586eaa15fc89858df37cfc00f96a61e2b13f97838d1d3ed95228b66" Oct 30 02:18:00.325438 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.325408 6221 generic.go:356] "Generic (PLEG): container finished" podID="1c9548ee-90ef-4fe2-9412-4da31c20dfba" containerID="d828962be39b77681f25d73fdb539631b9c68dad4027b7262ee0cff69484f110" exitCode=0 Oct 30 02:18:00.325894 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.325479 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv" event={"ID":"1c9548ee-90ef-4fe2-9412-4da31c20dfba","Type":"ContainerDied","Data":"d828962be39b77681f25d73fdb539631b9c68dad4027b7262ee0cff69484f110"} Oct 30 02:18:00.327468 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.327445 6221 generic.go:356] "Generic (PLEG): container finished" podID="61c55830-d642-4daa-8692-5f4f409b9a1a" containerID="517da38a773a77e45280517c2ed45aaf470a5296a9e26718e796ab3dc4858c42" exitCode=0 Oct 30 02:18:00.327630 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.327521 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd" event={"ID":"61c55830-d642-4daa-8692-5f4f409b9a1a","Type":"ContainerDied","Data":"517da38a773a77e45280517c2ed45aaf470a5296a9e26718e796ab3dc4858c42"} Oct 30 02:18:00.399477 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.399449 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv" Oct 30 02:18:00.446263 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.446234 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd" Oct 30 02:18:00.483991 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.483952 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c9548ee-90ef-4fe2-9412-4da31c20dfba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c9548ee-90ef-4fe2-9412-4da31c20dfba-workload-socket\") pod \"1c9548ee-90ef-4fe2-9412-4da31c20dfba\" (UID: \"1c9548ee-90ef-4fe2-9412-4da31c20dfba\") " Oct 30 02:18:00.484210 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.484009 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c9548ee-90ef-4fe2-9412-4da31c20dfba-kube-api-access-hnj9w\" (UniqueName: \"kubernetes.io/projected/1c9548ee-90ef-4fe2-9412-4da31c20dfba-kube-api-access-hnj9w\") pod \"1c9548ee-90ef-4fe2-9412-4da31c20dfba\" (UID: \"1c9548ee-90ef-4fe2-9412-4da31c20dfba\") " Oct 30 02:18:00.484210 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.484091 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c9548ee-90ef-4fe2-9412-4da31c20dfba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c9548ee-90ef-4fe2-9412-4da31c20dfba-istio-podinfo\") pod \"1c9548ee-90ef-4fe2-9412-4da31c20dfba\" (UID: \"1c9548ee-90ef-4fe2-9412-4da31c20dfba\") " Oct 30 02:18:00.484210 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.484160 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c9548ee-90ef-4fe2-9412-4da31c20dfba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c9548ee-90ef-4fe2-9412-4da31c20dfba-istio-envoy\") pod \"1c9548ee-90ef-4fe2-9412-4da31c20dfba\" (UID: \"1c9548ee-90ef-4fe2-9412-4da31c20dfba\") " Oct 30 02:18:00.484427 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.484247 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c9548ee-90ef-4fe2-9412-4da31c20dfba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c9548ee-90ef-4fe2-9412-4da31c20dfba-istio-data\") pod \"1c9548ee-90ef-4fe2-9412-4da31c20dfba\" (UID: \"1c9548ee-90ef-4fe2-9412-4da31c20dfba\") " Oct 30 02:18:00.484427 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.484323 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c9548ee-90ef-4fe2-9412-4da31c20dfba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c9548ee-90ef-4fe2-9412-4da31c20dfba-workload-certs\") pod \"1c9548ee-90ef-4fe2-9412-4da31c20dfba\" (UID: \"1c9548ee-90ef-4fe2-9412-4da31c20dfba\") " Oct 30 02:18:00.484427 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.484396 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c9548ee-90ef-4fe2-9412-4da31c20dfba-workload-socket" pod "1c9548ee-90ef-4fe2-9412-4da31c20dfba" (UID: "1c9548ee-90ef-4fe2-9412-4da31c20dfba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:00.484427 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.484417 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c9548ee-90ef-4fe2-9412-4da31c20dfba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c9548ee-90ef-4fe2-9412-4da31c20dfba-istiod-ca-cert\") pod \"1c9548ee-90ef-4fe2-9412-4da31c20dfba\" (UID: \"1c9548ee-90ef-4fe2-9412-4da31c20dfba\") " Oct 30 02:18:00.484716 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.484524 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c9548ee-90ef-4fe2-9412-4da31c20dfba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c9548ee-90ef-4fe2-9412-4da31c20dfba-credential-socket\") pod \"1c9548ee-90ef-4fe2-9412-4da31c20dfba\" (UID: \"1c9548ee-90ef-4fe2-9412-4da31c20dfba\") " Oct 30 02:18:00.484716 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.484575 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c9548ee-90ef-4fe2-9412-4da31c20dfba-istio-token\" (UniqueName: \"kubernetes.io/projected/1c9548ee-90ef-4fe2-9412-4da31c20dfba-istio-token\") pod \"1c9548ee-90ef-4fe2-9412-4da31c20dfba\" (UID: \"1c9548ee-90ef-4fe2-9412-4da31c20dfba\") " Oct 30 02:18:00.484835 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.484725 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c9548ee-90ef-4fe2-9412-4da31c20dfba-istio-data" pod "1c9548ee-90ef-4fe2-9412-4da31c20dfba" (UID: "1c9548ee-90ef-4fe2-9412-4da31c20dfba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:00.485098 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.485076 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c9548ee-90ef-4fe2-9412-4da31c20dfba-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:18:00.485180 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.485104 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c9548ee-90ef-4fe2-9412-4da31c20dfba-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:18:00.485359 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.485336 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c9548ee-90ef-4fe2-9412-4da31c20dfba-workload-certs" pod "1c9548ee-90ef-4fe2-9412-4da31c20dfba" (UID: "1c9548ee-90ef-4fe2-9412-4da31c20dfba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:00.486314 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.485681 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c9548ee-90ef-4fe2-9412-4da31c20dfba-istiod-ca-cert" pod "1c9548ee-90ef-4fe2-9412-4da31c20dfba" (UID: "1c9548ee-90ef-4fe2-9412-4da31c20dfba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:18:00.486314 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.485736 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c9548ee-90ef-4fe2-9412-4da31c20dfba-credential-socket" pod "1c9548ee-90ef-4fe2-9412-4da31c20dfba" (UID: "1c9548ee-90ef-4fe2-9412-4da31c20dfba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:00.487914 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.487889 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c9548ee-90ef-4fe2-9412-4da31c20dfba-istio-token" pod "1c9548ee-90ef-4fe2-9412-4da31c20dfba" (UID: "1c9548ee-90ef-4fe2-9412-4da31c20dfba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:00.488485 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.488451 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c9548ee-90ef-4fe2-9412-4da31c20dfba-istio-podinfo" pod "1c9548ee-90ef-4fe2-9412-4da31c20dfba" (UID: "1c9548ee-90ef-4fe2-9412-4da31c20dfba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:18:00.489021 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.488897 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c9548ee-90ef-4fe2-9412-4da31c20dfba-istio-envoy" pod "1c9548ee-90ef-4fe2-9412-4da31c20dfba" (UID: "1c9548ee-90ef-4fe2-9412-4da31c20dfba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:00.489515 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.489484 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c9548ee-90ef-4fe2-9412-4da31c20dfba-kube-api-access-hnj9w" pod "1c9548ee-90ef-4fe2-9412-4da31c20dfba" (UID: "1c9548ee-90ef-4fe2-9412-4da31c20dfba"). InnerVolumeSpecName "kube-api-access-hnj9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:00.585797 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.585762 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61c55830-d642-4daa-8692-5f4f409b9a1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c55830-d642-4daa-8692-5f4f409b9a1a-workload-socket\") pod \"61c55830-d642-4daa-8692-5f4f409b9a1a\" (UID: \"61c55830-d642-4daa-8692-5f4f409b9a1a\") " Oct 30 02:18:00.586000 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.585812 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61c55830-d642-4daa-8692-5f4f409b9a1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61c55830-d642-4daa-8692-5f4f409b9a1a-istio-envoy\") pod \"61c55830-d642-4daa-8692-5f4f409b9a1a\" (UID: \"61c55830-d642-4daa-8692-5f4f409b9a1a\") " Oct 30 02:18:00.586000 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.585845 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61c55830-d642-4daa-8692-5f4f409b9a1a-kube-api-access-vrnt8\" (UniqueName: \"kubernetes.io/projected/61c55830-d642-4daa-8692-5f4f409b9a1a-kube-api-access-vrnt8\") pod \"61c55830-d642-4daa-8692-5f4f409b9a1a\" (UID: \"61c55830-d642-4daa-8692-5f4f409b9a1a\") " Oct 30 02:18:00.586000 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.585886 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61c55830-d642-4daa-8692-5f4f409b9a1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61c55830-d642-4daa-8692-5f4f409b9a1a-istiod-ca-cert\") pod \"61c55830-d642-4daa-8692-5f4f409b9a1a\" (UID: \"61c55830-d642-4daa-8692-5f4f409b9a1a\") " Oct 30 02:18:00.586000 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.585913 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61c55830-d642-4daa-8692-5f4f409b9a1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61c55830-d642-4daa-8692-5f4f409b9a1a-workload-certs\") pod \"61c55830-d642-4daa-8692-5f4f409b9a1a\" (UID: \"61c55830-d642-4daa-8692-5f4f409b9a1a\") " Oct 30 02:18:00.586000 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.585969 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61c55830-d642-4daa-8692-5f4f409b9a1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61c55830-d642-4daa-8692-5f4f409b9a1a-istio-podinfo\") pod \"61c55830-d642-4daa-8692-5f4f409b9a1a\" (UID: \"61c55830-d642-4daa-8692-5f4f409b9a1a\") " Oct 30 02:18:00.586285 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.586025 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61c55830-d642-4daa-8692-5f4f409b9a1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c55830-d642-4daa-8692-5f4f409b9a1a-credential-socket\") pod \"61c55830-d642-4daa-8692-5f4f409b9a1a\" (UID: \"61c55830-d642-4daa-8692-5f4f409b9a1a\") " Oct 30 02:18:00.586285 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.586077 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61c55830-d642-4daa-8692-5f4f409b9a1a-istio-token\" (UniqueName: \"kubernetes.io/projected/61c55830-d642-4daa-8692-5f4f409b9a1a-istio-token\") pod \"61c55830-d642-4daa-8692-5f4f409b9a1a\" (UID: \"61c55830-d642-4daa-8692-5f4f409b9a1a\") " Oct 30 02:18:00.586285 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.586107 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61c55830-d642-4daa-8692-5f4f409b9a1a-workload-socket" pod "61c55830-d642-4daa-8692-5f4f409b9a1a" (UID: "61c55830-d642-4daa-8692-5f4f409b9a1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:00.586285 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.586177 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61c55830-d642-4daa-8692-5f4f409b9a1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61c55830-d642-4daa-8692-5f4f409b9a1a-istio-data\") pod \"61c55830-d642-4daa-8692-5f4f409b9a1a\" (UID: \"61c55830-d642-4daa-8692-5f4f409b9a1a\") " Oct 30 02:18:00.586484 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.586319 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61c55830-d642-4daa-8692-5f4f409b9a1a-workload-certs" pod "61c55830-d642-4daa-8692-5f4f409b9a1a" (UID: "61c55830-d642-4daa-8692-5f4f409b9a1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:00.586484 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.586356 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61c55830-d642-4daa-8692-5f4f409b9a1a-credential-socket" pod "61c55830-d642-4daa-8692-5f4f409b9a1a" (UID: "61c55830-d642-4daa-8692-5f4f409b9a1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:00.586484 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.586399 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61c55830-d642-4daa-8692-5f4f409b9a1a-istiod-ca-cert" pod "61c55830-d642-4daa-8692-5f4f409b9a1a" (UID: "61c55830-d642-4daa-8692-5f4f409b9a1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:18:00.586651 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.586605 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61c55830-d642-4daa-8692-5f4f409b9a1a-istio-data" pod "61c55830-d642-4daa-8692-5f4f409b9a1a" (UID: "61c55830-d642-4daa-8692-5f4f409b9a1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:00.586833 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.586814 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c9548ee-90ef-4fe2-9412-4da31c20dfba-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:18:00.586915 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.586835 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c9548ee-90ef-4fe2-9412-4da31c20dfba-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:18:00.586915 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.586844 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61c55830-d642-4daa-8692-5f4f409b9a1a-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:18:00.586915 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.586852 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnj9w\" (UniqueName: \"kubernetes.io/projected/1c9548ee-90ef-4fe2-9412-4da31c20dfba-kube-api-access-hnj9w\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:18:00.586915 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.586861 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c9548ee-90ef-4fe2-9412-4da31c20dfba-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:18:00.586915 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.586869 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c55830-d642-4daa-8692-5f4f409b9a1a-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:18:00.586915 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.586876 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c9548ee-90ef-4fe2-9412-4da31c20dfba-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:18:00.586915 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.586885 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61c55830-d642-4daa-8692-5f4f409b9a1a-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:18:00.586915 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.586893 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61c55830-d642-4daa-8692-5f4f409b9a1a-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:18:00.586915 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.586902 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c9548ee-90ef-4fe2-9412-4da31c20dfba-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:18:00.586915 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.586911 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61c55830-d642-4daa-8692-5f4f409b9a1a-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:18:00.586915 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.586920 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c9548ee-90ef-4fe2-9412-4da31c20dfba-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:18:00.588220 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.588201 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61c55830-d642-4daa-8692-5f4f409b9a1a-istio-envoy" pod "61c55830-d642-4daa-8692-5f4f409b9a1a" (UID: "61c55830-d642-4daa-8692-5f4f409b9a1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:18:00.588480 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.588465 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61c55830-d642-4daa-8692-5f4f409b9a1a-istio-token" pod "61c55830-d642-4daa-8692-5f4f409b9a1a" (UID: "61c55830-d642-4daa-8692-5f4f409b9a1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:00.588661 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.588641 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61c55830-d642-4daa-8692-5f4f409b9a1a-istio-podinfo" pod "61c55830-d642-4daa-8692-5f4f409b9a1a" (UID: "61c55830-d642-4daa-8692-5f4f409b9a1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:18:00.588711 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.588665 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61c55830-d642-4daa-8692-5f4f409b9a1a-kube-api-access-vrnt8" pod "61c55830-d642-4daa-8692-5f4f409b9a1a" (UID: "61c55830-d642-4daa-8692-5f4f409b9a1a"). InnerVolumeSpecName "kube-api-access-vrnt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:00.688747 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.688668 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61c55830-d642-4daa-8692-5f4f409b9a1a-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:18:00.688747 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.688695 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61c55830-d642-4daa-8692-5f4f409b9a1a-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:18:00.688747 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.688705 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrnt8\" (UniqueName: \"kubernetes.io/projected/61c55830-d642-4daa-8692-5f4f409b9a1a-kube-api-access-vrnt8\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:18:00.688747 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:00.688715 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61c55830-d642-4daa-8692-5f4f409b9a1a-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:18:01.335268 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:01.335231 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv" Oct 30 02:18:01.335268 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:01.335248 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv" event={"ID":"1c9548ee-90ef-4fe2-9412-4da31c20dfba","Type":"ContainerDied","Data":"e8a3dd0c455d1901edc3281760a37b2fd190b381469fa0a27765f58735efe955"} Oct 30 02:18:01.335840 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:01.335349 6221 scope.go:122] "RemoveContainer" containerID="d828962be39b77681f25d73fdb539631b9c68dad4027b7262ee0cff69484f110" Oct 30 02:18:01.337149 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:01.336894 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd" event={"ID":"61c55830-d642-4daa-8692-5f4f409b9a1a","Type":"ContainerDied","Data":"c88c7a93ca9df4a54f9319fd2d2ec383530a012954a8892add44d58dc42d1555"} Oct 30 02:18:01.337149 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:01.336953 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd" Oct 30 02:18:01.346475 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:01.346454 6221 scope.go:122] "RemoveContainer" containerID="517da38a773a77e45280517c2ed45aaf470a5296a9e26718e796ab3dc4858c42" Oct 30 02:18:01.401732 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:01.401646 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv"] Oct 30 02:18:01.413979 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:01.413946 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-j6rcv"] Oct 30 02:18:01.450743 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:01.450714 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd"] Oct 30 02:18:01.453729 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:01.453703 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-601d4b-openshift-default-66855f7959-8wbtd"] Oct 30 02:18:02.547437 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:02.547407 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c9548ee-90ef-4fe2-9412-4da31c20dfba" path="/var/lib/kubelet/pods/1c9548ee-90ef-4fe2-9412-4da31c20dfba/volumes" Oct 30 02:18:02.547953 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:02.547937 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61c55830-d642-4daa-8692-5f4f409b9a1a" path="/var/lib/kubelet/pods/61c55830-d642-4daa-8692-5f4f409b9a1a/volumes" Oct 30 02:18:06.819937 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:06.819902 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-093d23-5fd64fb8d4-z266p"] Oct 30 02:18:11.567925 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:11.567836 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-eda881-74bfc54cbd-4tc64"] Oct 30 02:18:11.568467 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:11.568362 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eda881-74bfc54cbd-4tc64" podUID="c26c4344-4ccb-44d9-ad1d-cfd8720bc56d" containerName="maas-api" containerID="cri-o://76e28a44d9968b949b498442b4929f86a7d9634b0b960b7fcc938f2a92876385" gracePeriod=30 Oct 30 02:18:11.820768 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:11.820318 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-093d23-5fd64fb8d4-z266p" podUID="ab3bacd1-4d0c-4c6b-b672-582c8d729702" containerName="payload-processing" containerID="cri-o://8cd2a22f2763c98a2c36ba8b6919a437e82cc0a632e323737e7b0e3b5d705cb7" gracePeriod=25 Oct 30 02:18:11.861058 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:11.861032 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eda881-74bfc54cbd-4tc64" Oct 30 02:18:11.973107 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:11.973067 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c26c4344-4ccb-44d9-ad1d-cfd8720bc56d-kube-api-access-98qmf\" (UniqueName: \"kubernetes.io/projected/c26c4344-4ccb-44d9-ad1d-cfd8720bc56d-kube-api-access-98qmf\") pod \"c26c4344-4ccb-44d9-ad1d-cfd8720bc56d\" (UID: \"c26c4344-4ccb-44d9-ad1d-cfd8720bc56d\") " Oct 30 02:18:11.973290 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:11.973253 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c26c4344-4ccb-44d9-ad1d-cfd8720bc56d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c26c4344-4ccb-44d9-ad1d-cfd8720bc56d-openshift-service-ca\") pod \"c26c4344-4ccb-44d9-ad1d-cfd8720bc56d\" (UID: \"c26c4344-4ccb-44d9-ad1d-cfd8720bc56d\") " Oct 30 02:18:11.973356 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:11.973319 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c26c4344-4ccb-44d9-ad1d-cfd8720bc56d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c26c4344-4ccb-44d9-ad1d-cfd8720bc56d-maas-api-tls\") pod \"c26c4344-4ccb-44d9-ad1d-cfd8720bc56d\" (UID: \"c26c4344-4ccb-44d9-ad1d-cfd8720bc56d\") " Oct 30 02:18:11.973455 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:11.973438 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c26c4344-4ccb-44d9-ad1d-cfd8720bc56d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c26c4344-4ccb-44d9-ad1d-cfd8720bc56d-metrics-certs\") pod \"c26c4344-4ccb-44d9-ad1d-cfd8720bc56d\" (UID: \"c26c4344-4ccb-44d9-ad1d-cfd8720bc56d\") " Oct 30 02:18:11.973719 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:11.973685 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c26c4344-4ccb-44d9-ad1d-cfd8720bc56d-openshift-service-ca" pod "c26c4344-4ccb-44d9-ad1d-cfd8720bc56d" (UID: "c26c4344-4ccb-44d9-ad1d-cfd8720bc56d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:18:11.973862 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:11.973842 6221 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c26c4344-4ccb-44d9-ad1d-cfd8720bc56d-openshift-service-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:18:11.975972 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:11.975941 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c26c4344-4ccb-44d9-ad1d-cfd8720bc56d-metrics-certs" pod "c26c4344-4ccb-44d9-ad1d-cfd8720bc56d" (UID: "c26c4344-4ccb-44d9-ad1d-cfd8720bc56d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:18:11.976938 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:11.976912 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c26c4344-4ccb-44d9-ad1d-cfd8720bc56d-kube-api-access-98qmf" pod "c26c4344-4ccb-44d9-ad1d-cfd8720bc56d" (UID: "c26c4344-4ccb-44d9-ad1d-cfd8720bc56d"). InnerVolumeSpecName "kube-api-access-98qmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:11.978098 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:11.978071 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c26c4344-4ccb-44d9-ad1d-cfd8720bc56d-maas-api-tls" pod "c26c4344-4ccb-44d9-ad1d-cfd8720bc56d" (UID: "c26c4344-4ccb-44d9-ad1d-cfd8720bc56d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:18:12.075057 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:12.074979 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98qmf\" (UniqueName: \"kubernetes.io/projected/c26c4344-4ccb-44d9-ad1d-cfd8720bc56d-kube-api-access-98qmf\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:18:12.075057 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:12.075009 6221 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c26c4344-4ccb-44d9-ad1d-cfd8720bc56d-maas-api-tls\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:18:12.075057 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:12.075018 6221 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c26c4344-4ccb-44d9-ad1d-cfd8720bc56d-metrics-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:18:12.406767 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:12.406682 6221 generic.go:356] "Generic (PLEG): container finished" podID="c26c4344-4ccb-44d9-ad1d-cfd8720bc56d" containerID="76e28a44d9968b949b498442b4929f86a7d9634b0b960b7fcc938f2a92876385" exitCode=0 Oct 30 02:18:12.406767 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:12.406730 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eda881-74bfc54cbd-4tc64" Oct 30 02:18:12.406767 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:12.406741 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eda881-74bfc54cbd-4tc64" event={"ID":"c26c4344-4ccb-44d9-ad1d-cfd8720bc56d","Type":"ContainerDied","Data":"76e28a44d9968b949b498442b4929f86a7d9634b0b960b7fcc938f2a92876385"} Oct 30 02:18:12.407004 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:12.406829 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-eda881-74bfc54cbd-4tc64" event={"ID":"c26c4344-4ccb-44d9-ad1d-cfd8720bc56d","Type":"ContainerDied","Data":"c89b1f604699e9ad7be29e3eae1c1b69aa9bd687b0ee76a3dc0e4b4adb31d720"} Oct 30 02:18:12.407004 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:12.406861 6221 scope.go:122] "RemoveContainer" containerID="76e28a44d9968b949b498442b4929f86a7d9634b0b960b7fcc938f2a92876385" Oct 30 02:18:12.418231 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:12.418214 6221 scope.go:122] "RemoveContainer" containerID="76e28a44d9968b949b498442b4929f86a7d9634b0b960b7fcc938f2a92876385" Oct 30 02:18:12.418505 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:18:12.418483 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76e28a44d9968b949b498442b4929f86a7d9634b0b960b7fcc938f2a92876385\": container with ID starting with 76e28a44d9968b949b498442b4929f86a7d9634b0b960b7fcc938f2a92876385 not found: ID does not exist" containerID="76e28a44d9968b949b498442b4929f86a7d9634b0b960b7fcc938f2a92876385" Oct 30 02:18:12.418587 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:12.418514 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76e28a44d9968b949b498442b4929f86a7d9634b0b960b7fcc938f2a92876385"} err="failed to get container status \"76e28a44d9968b949b498442b4929f86a7d9634b0b960b7fcc938f2a92876385\": rpc error: code = NotFound desc = could not find container \"76e28a44d9968b949b498442b4929f86a7d9634b0b960b7fcc938f2a92876385\": container with ID starting with 76e28a44d9968b949b498442b4929f86a7d9634b0b960b7fcc938f2a92876385 not found: ID does not exist" Oct 30 02:18:12.458776 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:12.458741 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-eda881-74bfc54cbd-4tc64"] Oct 30 02:18:12.471183 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:12.471151 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-eda881-74bfc54cbd-4tc64"] Oct 30 02:18:12.547936 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:12.547905 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c26c4344-4ccb-44d9-ad1d-cfd8720bc56d" path="/var/lib/kubelet/pods/c26c4344-4ccb-44d9-ad1d-cfd8720bc56d/volumes" Oct 30 02:18:17.374121 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:17.374082 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-093d23-5fd64fb8d4-z266p" podUID="ab3bacd1-4d0c-4c6b-b672-582c8d729702" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:18:18.204760 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:18.204724 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-093d23-55b74987dd-8nwrk"] Oct 30 02:18:18.205020 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:18.204989 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-093d23-55b74987dd-8nwrk" podUID="08f6b864-02b6-4aa9-8616-777fb088d0a4" containerName="maas-api" containerID="cri-o://9712d9e65affa092ac889d7137c0d6cdd73586382cb5fd9323f9d67f4e796c93" gracePeriod=30 Oct 30 02:18:18.498456 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:18.496764 6221 generic.go:356] "Generic (PLEG): container finished" podID="08f6b864-02b6-4aa9-8616-777fb088d0a4" containerID="9712d9e65affa092ac889d7137c0d6cdd73586382cb5fd9323f9d67f4e796c93" exitCode=0 Oct 30 02:18:18.498456 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:18.496990 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-093d23-55b74987dd-8nwrk" event={"ID":"08f6b864-02b6-4aa9-8616-777fb088d0a4","Type":"ContainerDied","Data":"9712d9e65affa092ac889d7137c0d6cdd73586382cb5fd9323f9d67f4e796c93"} Oct 30 02:18:18.593226 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:18.593193 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-093d23-55b74987dd-8nwrk" Oct 30 02:18:18.675603 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:18.675568 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08f6b864-02b6-4aa9-8616-777fb088d0a4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08f6b864-02b6-4aa9-8616-777fb088d0a4-maas-api-tls\") pod \"08f6b864-02b6-4aa9-8616-777fb088d0a4\" (UID: \"08f6b864-02b6-4aa9-8616-777fb088d0a4\") " Oct 30 02:18:18.675782 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:18.675654 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08f6b864-02b6-4aa9-8616-777fb088d0a4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08f6b864-02b6-4aa9-8616-777fb088d0a4-openshift-service-ca\") pod \"08f6b864-02b6-4aa9-8616-777fb088d0a4\" (UID: \"08f6b864-02b6-4aa9-8616-777fb088d0a4\") " Oct 30 02:18:18.675782 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:18.675745 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08f6b864-02b6-4aa9-8616-777fb088d0a4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/08f6b864-02b6-4aa9-8616-777fb088d0a4-metrics-certs\") pod \"08f6b864-02b6-4aa9-8616-777fb088d0a4\" (UID: \"08f6b864-02b6-4aa9-8616-777fb088d0a4\") " Oct 30 02:18:18.675874 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:18.675784 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08f6b864-02b6-4aa9-8616-777fb088d0a4-kube-api-access-77g8j\" (UniqueName: \"kubernetes.io/projected/08f6b864-02b6-4aa9-8616-777fb088d0a4-kube-api-access-77g8j\") pod \"08f6b864-02b6-4aa9-8616-777fb088d0a4\" (UID: \"08f6b864-02b6-4aa9-8616-777fb088d0a4\") " Oct 30 02:18:18.676106 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:18.676069 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08f6b864-02b6-4aa9-8616-777fb088d0a4-openshift-service-ca" pod "08f6b864-02b6-4aa9-8616-777fb088d0a4" (UID: "08f6b864-02b6-4aa9-8616-777fb088d0a4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:18:18.676366 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:18.676346 6221 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08f6b864-02b6-4aa9-8616-777fb088d0a4-openshift-service-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:18:18.677954 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:18.677931 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08f6b864-02b6-4aa9-8616-777fb088d0a4-kube-api-access-77g8j" pod "08f6b864-02b6-4aa9-8616-777fb088d0a4" (UID: "08f6b864-02b6-4aa9-8616-777fb088d0a4"). InnerVolumeSpecName "kube-api-access-77g8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:18.678057 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:18.678024 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08f6b864-02b6-4aa9-8616-777fb088d0a4-metrics-certs" pod "08f6b864-02b6-4aa9-8616-777fb088d0a4" (UID: "08f6b864-02b6-4aa9-8616-777fb088d0a4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:18:18.678178 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:18.678161 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08f6b864-02b6-4aa9-8616-777fb088d0a4-maas-api-tls" pod "08f6b864-02b6-4aa9-8616-777fb088d0a4" (UID: "08f6b864-02b6-4aa9-8616-777fb088d0a4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:18:18.777284 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:18.777189 6221 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08f6b864-02b6-4aa9-8616-777fb088d0a4-maas-api-tls\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:18:18.777284 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:18.777221 6221 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08f6b864-02b6-4aa9-8616-777fb088d0a4-metrics-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:18:18.777284 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:18.777234 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77g8j\" (UniqueName: \"kubernetes.io/projected/08f6b864-02b6-4aa9-8616-777fb088d0a4-kube-api-access-77g8j\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:18:19.512559 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:19.511739 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-093d23-55b74987dd-8nwrk" event={"ID":"08f6b864-02b6-4aa9-8616-777fb088d0a4","Type":"ContainerDied","Data":"75fc19b6c13106a24e238550939c77e36171da17385af5fa140c0958a41ea828"} Oct 30 02:18:19.512559 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:19.511838 6221 scope.go:122] "RemoveContainer" containerID="9712d9e65affa092ac889d7137c0d6cdd73586382cb5fd9323f9d67f4e796c93" Oct 30 02:18:19.512559 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:19.511772 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-093d23-55b74987dd-8nwrk" Oct 30 02:18:19.564402 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:19.564366 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-093d23-55b74987dd-8nwrk"] Oct 30 02:18:19.576391 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:19.576362 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-093d23-55b74987dd-8nwrk"] Oct 30 02:18:20.564156 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:20.564115 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08f6b864-02b6-4aa9-8616-777fb088d0a4" path="/var/lib/kubelet/pods/08f6b864-02b6-4aa9-8616-777fb088d0a4/volumes" Oct 30 02:18:21.540939 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:18:21.540892 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 02:18:27.374295 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:27.374251 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-093d23-5fd64fb8d4-z266p" podUID="ab3bacd1-4d0c-4c6b-b672-582c8d729702" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:18:30.076687 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:30.076645 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 02:18:32.113414 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:32.113385 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-093d23-5fd64fb8d4-z266p" Oct 30 02:18:32.144621 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:32.144592 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab3bacd1-4d0c-4c6b-b672-582c8d729702-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab3bacd1-4d0c-4c6b-b672-582c8d729702-plugins-config-volume\") pod \"ab3bacd1-4d0c-4c6b-b672-582c8d729702\" (UID: \"ab3bacd1-4d0c-4c6b-b672-582c8d729702\") " Oct 30 02:18:32.144768 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:32.144695 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab3bacd1-4d0c-4c6b-b672-582c8d729702-kube-api-access-hz92k\" (UniqueName: \"kubernetes.io/projected/ab3bacd1-4d0c-4c6b-b672-582c8d729702-kube-api-access-hz92k\") pod \"ab3bacd1-4d0c-4c6b-b672-582c8d729702\" (UID: \"ab3bacd1-4d0c-4c6b-b672-582c8d729702\") " Oct 30 02:18:32.144967 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:32.144944 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab3bacd1-4d0c-4c6b-b672-582c8d729702-plugins-config-volume" pod "ab3bacd1-4d0c-4c6b-b672-582c8d729702" (UID: "ab3bacd1-4d0c-4c6b-b672-582c8d729702"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:18:32.145112 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:32.145092 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab3bacd1-4d0c-4c6b-b672-582c8d729702-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:18:32.146815 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:32.146794 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab3bacd1-4d0c-4c6b-b672-582c8d729702-kube-api-access-hz92k" pod "ab3bacd1-4d0c-4c6b-b672-582c8d729702" (UID: "ab3bacd1-4d0c-4c6b-b672-582c8d729702"). InnerVolumeSpecName "kube-api-access-hz92k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:32.246681 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:32.246643 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz92k\" (UniqueName: \"kubernetes.io/projected/ab3bacd1-4d0c-4c6b-b672-582c8d729702-kube-api-access-hz92k\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:18:32.331394 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:32.331355 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2e618f-7975b944f8-8f6dz"] Oct 30 02:18:32.365228 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:32.365143 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2e618f-f475b8c6c-lr97t"] Oct 30 02:18:32.602772 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:32.602737 6221 generic.go:356] "Generic (PLEG): container finished" podID="ab3bacd1-4d0c-4c6b-b672-582c8d729702" containerID="8cd2a22f2763c98a2c36ba8b6919a437e82cc0a632e323737e7b0e3b5d705cb7" exitCode=0 Oct 30 02:18:32.602952 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:32.602781 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-093d23-5fd64fb8d4-z266p" Oct 30 02:18:32.602952 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:32.602795 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-093d23-5fd64fb8d4-z266p" event={"ID":"ab3bacd1-4d0c-4c6b-b672-582c8d729702","Type":"ContainerDied","Data":"8cd2a22f2763c98a2c36ba8b6919a437e82cc0a632e323737e7b0e3b5d705cb7"} Oct 30 02:18:32.602952 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:32.602850 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-093d23-5fd64fb8d4-z266p" event={"ID":"ab3bacd1-4d0c-4c6b-b672-582c8d729702","Type":"ContainerDied","Data":"6ad6c1bba46d3aaf6fed6cfad22572898db799069ad4c4d3c7f5aed2f73e32f7"} Oct 30 02:18:32.602952 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:32.602871 6221 scope.go:122] "RemoveContainer" containerID="8cd2a22f2763c98a2c36ba8b6919a437e82cc0a632e323737e7b0e3b5d705cb7" Oct 30 02:18:32.612840 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:32.612819 6221 scope.go:122] "RemoveContainer" containerID="8cd2a22f2763c98a2c36ba8b6919a437e82cc0a632e323737e7b0e3b5d705cb7" Oct 30 02:18:32.613143 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:18:32.613121 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cd2a22f2763c98a2c36ba8b6919a437e82cc0a632e323737e7b0e3b5d705cb7\": container with ID starting with 8cd2a22f2763c98a2c36ba8b6919a437e82cc0a632e323737e7b0e3b5d705cb7 not found: ID does not exist" containerID="8cd2a22f2763c98a2c36ba8b6919a437e82cc0a632e323737e7b0e3b5d705cb7" Oct 30 02:18:32.613198 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:32.613153 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cd2a22f2763c98a2c36ba8b6919a437e82cc0a632e323737e7b0e3b5d705cb7"} err="failed to get container status \"8cd2a22f2763c98a2c36ba8b6919a437e82cc0a632e323737e7b0e3b5d705cb7\": rpc error: code = NotFound desc = could not find container \"8cd2a22f2763c98a2c36ba8b6919a437e82cc0a632e323737e7b0e3b5d705cb7\": container with ID starting with 8cd2a22f2763c98a2c36ba8b6919a437e82cc0a632e323737e7b0e3b5d705cb7 not found: ID does not exist" Oct 30 02:18:32.656203 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:32.656129 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-093d23-5fd64fb8d4-z266p"] Oct 30 02:18:32.660764 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:32.660737 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-093d23-5fd64fb8d4-z266p"] Oct 30 02:18:34.548079 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:34.548042 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab3bacd1-4d0c-4c6b-b672-582c8d729702" path="/var/lib/kubelet/pods/ab3bacd1-4d0c-4c6b-b672-582c8d729702/volumes" Oct 30 02:18:37.331910 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:37.331867 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-2e618f-7975b944f8-8f6dz" podUID="16bdf204-496b-4e72-b5cd-e7b5719043e5" containerName="payload-processing" containerID="cri-o://f63350ef310c4c3a1709316f216f429d91a63cfaada4102d24d41133770042a5" gracePeriod=25 Oct 30 02:18:37.365627 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:37.365580 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2e618f-f475b8c6c-lr97t" podUID="fd5ba5d9-23ee-42d4-97f0-3224353b90ec" containerName="payload-pre-processing" containerID="cri-o://a161132d7da7853c90ed85354f27f2bd7f2ad298019d4667560d3e7bcbdd7651" gracePeriod=25 Oct 30 02:18:42.216625 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:42.216583 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2e618f-f475b8c6c-lr97t" podUID="fd5ba5d9-23ee-42d4-97f0-3224353b90ec" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:18:42.217101 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:42.216587 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-2e618f-7975b944f8-8f6dz" podUID="16bdf204-496b-4e72-b5cd-e7b5719043e5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:18:51.494142 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:18:51.494104 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 02:18:52.215574 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:52.215513 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-2e618f-7975b944f8-8f6dz" podUID="16bdf204-496b-4e72-b5cd-e7b5719043e5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:18:52.215751 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:52.215513 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2e618f-f475b8c6c-lr97t" podUID="fd5ba5d9-23ee-42d4-97f0-3224353b90ec" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:18:57.650584 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:57.650549 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-2e618f-7975b944f8-8f6dz" Oct 30 02:18:57.671084 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:57.671063 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2e618f-f475b8c6c-lr97t" Oct 30 02:18:57.762377 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:57.762344 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16bdf204-496b-4e72-b5cd-e7b5719043e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16bdf204-496b-4e72-b5cd-e7b5719043e5-plugins-config-volume\") pod \"16bdf204-496b-4e72-b5cd-e7b5719043e5\" (UID: \"16bdf204-496b-4e72-b5cd-e7b5719043e5\") " Oct 30 02:18:57.762627 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:57.762411 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16bdf204-496b-4e72-b5cd-e7b5719043e5-kube-api-access-vwx2j\" (UniqueName: \"kubernetes.io/projected/16bdf204-496b-4e72-b5cd-e7b5719043e5-kube-api-access-vwx2j\") pod \"16bdf204-496b-4e72-b5cd-e7b5719043e5\" (UID: \"16bdf204-496b-4e72-b5cd-e7b5719043e5\") " Oct 30 02:18:57.762796 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:57.762752 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16bdf204-496b-4e72-b5cd-e7b5719043e5-plugins-config-volume" pod "16bdf204-496b-4e72-b5cd-e7b5719043e5" (UID: "16bdf204-496b-4e72-b5cd-e7b5719043e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:18:57.762971 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:57.762951 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16bdf204-496b-4e72-b5cd-e7b5719043e5-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:18:57.764203 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:57.764174 6221 generic.go:356] "Generic (PLEG): container finished" podID="fd5ba5d9-23ee-42d4-97f0-3224353b90ec" containerID="a161132d7da7853c90ed85354f27f2bd7f2ad298019d4667560d3e7bcbdd7651" exitCode=0 Oct 30 02:18:57.764329 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:57.764226 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2e618f-f475b8c6c-lr97t" Oct 30 02:18:57.764329 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:57.764216 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2e618f-f475b8c6c-lr97t" event={"ID":"fd5ba5d9-23ee-42d4-97f0-3224353b90ec","Type":"ContainerDied","Data":"a161132d7da7853c90ed85354f27f2bd7f2ad298019d4667560d3e7bcbdd7651"} Oct 30 02:18:57.764459 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:57.764360 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-2e618f-f475b8c6c-lr97t" event={"ID":"fd5ba5d9-23ee-42d4-97f0-3224353b90ec","Type":"ContainerDied","Data":"c7fe03d12dadd73019bc6f8da2b68919e6b42e6152a8cf3fc0bf9526fd5a9e5e"} Oct 30 02:18:57.764459 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:57.764377 6221 scope.go:122] "RemoveContainer" containerID="a161132d7da7853c90ed85354f27f2bd7f2ad298019d4667560d3e7bcbdd7651" Oct 30 02:18:57.765010 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:57.764990 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16bdf204-496b-4e72-b5cd-e7b5719043e5-kube-api-access-vwx2j" pod "16bdf204-496b-4e72-b5cd-e7b5719043e5" (UID: "16bdf204-496b-4e72-b5cd-e7b5719043e5"). InnerVolumeSpecName "kube-api-access-vwx2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:57.766022 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:57.766000 6221 generic.go:356] "Generic (PLEG): container finished" podID="16bdf204-496b-4e72-b5cd-e7b5719043e5" containerID="f63350ef310c4c3a1709316f216f429d91a63cfaada4102d24d41133770042a5" exitCode=0 Oct 30 02:18:57.766144 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:57.766038 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-2e618f-7975b944f8-8f6dz" Oct 30 02:18:57.766144 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:57.766054 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-2e618f-7975b944f8-8f6dz" event={"ID":"16bdf204-496b-4e72-b5cd-e7b5719043e5","Type":"ContainerDied","Data":"f63350ef310c4c3a1709316f216f429d91a63cfaada4102d24d41133770042a5"} Oct 30 02:18:57.766144 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:57.766113 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-2e618f-7975b944f8-8f6dz" event={"ID":"16bdf204-496b-4e72-b5cd-e7b5719043e5","Type":"ContainerDied","Data":"a76b137db87dece91da6800f32a1aab90ff79811ddac48250c8b2287b1063869"} Oct 30 02:18:57.776367 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:57.776347 6221 scope.go:122] "RemoveContainer" containerID="a161132d7da7853c90ed85354f27f2bd7f2ad298019d4667560d3e7bcbdd7651" Oct 30 02:18:57.776665 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:18:57.776645 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a161132d7da7853c90ed85354f27f2bd7f2ad298019d4667560d3e7bcbdd7651\": container with ID starting with a161132d7da7853c90ed85354f27f2bd7f2ad298019d4667560d3e7bcbdd7651 not found: ID does not exist" containerID="a161132d7da7853c90ed85354f27f2bd7f2ad298019d4667560d3e7bcbdd7651" Oct 30 02:18:57.776727 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:57.776683 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a161132d7da7853c90ed85354f27f2bd7f2ad298019d4667560d3e7bcbdd7651"} err="failed to get container status \"a161132d7da7853c90ed85354f27f2bd7f2ad298019d4667560d3e7bcbdd7651\": rpc error: code = NotFound desc = could not find container \"a161132d7da7853c90ed85354f27f2bd7f2ad298019d4667560d3e7bcbdd7651\": container with ID starting with a161132d7da7853c90ed85354f27f2bd7f2ad298019d4667560d3e7bcbdd7651 not found: ID does not exist" Oct 30 02:18:57.776727 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:57.776700 6221 scope.go:122] "RemoveContainer" containerID="f63350ef310c4c3a1709316f216f429d91a63cfaada4102d24d41133770042a5" Oct 30 02:18:57.785912 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:57.785896 6221 scope.go:122] "RemoveContainer" containerID="f63350ef310c4c3a1709316f216f429d91a63cfaada4102d24d41133770042a5" Oct 30 02:18:57.786211 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:18:57.786191 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f63350ef310c4c3a1709316f216f429d91a63cfaada4102d24d41133770042a5\": container with ID starting with f63350ef310c4c3a1709316f216f429d91a63cfaada4102d24d41133770042a5 not found: ID does not exist" containerID="f63350ef310c4c3a1709316f216f429d91a63cfaada4102d24d41133770042a5" Oct 30 02:18:57.786269 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:57.786219 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f63350ef310c4c3a1709316f216f429d91a63cfaada4102d24d41133770042a5"} err="failed to get container status \"f63350ef310c4c3a1709316f216f429d91a63cfaada4102d24d41133770042a5\": rpc error: code = NotFound desc = could not find container \"f63350ef310c4c3a1709316f216f429d91a63cfaada4102d24d41133770042a5\": container with ID starting with f63350ef310c4c3a1709316f216f429d91a63cfaada4102d24d41133770042a5 not found: ID does not exist" Oct 30 02:18:57.820223 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:57.820202 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2e618f-7975b944f8-8f6dz"] Oct 30 02:18:57.825502 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:57.825475 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2e618f-7975b944f8-8f6dz"] Oct 30 02:18:57.863450 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:57.863396 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd5ba5d9-23ee-42d4-97f0-3224353b90ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd5ba5d9-23ee-42d4-97f0-3224353b90ec-plugins-config-volume\") pod \"fd5ba5d9-23ee-42d4-97f0-3224353b90ec\" (UID: \"fd5ba5d9-23ee-42d4-97f0-3224353b90ec\") " Oct 30 02:18:57.863560 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:57.863487 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd5ba5d9-23ee-42d4-97f0-3224353b90ec-kube-api-access-7zjbx\" (UniqueName: \"kubernetes.io/projected/fd5ba5d9-23ee-42d4-97f0-3224353b90ec-kube-api-access-7zjbx\") pod \"fd5ba5d9-23ee-42d4-97f0-3224353b90ec\" (UID: \"fd5ba5d9-23ee-42d4-97f0-3224353b90ec\") " Oct 30 02:18:57.863783 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:57.863761 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd5ba5d9-23ee-42d4-97f0-3224353b90ec-plugins-config-volume" pod "fd5ba5d9-23ee-42d4-97f0-3224353b90ec" (UID: "fd5ba5d9-23ee-42d4-97f0-3224353b90ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:18:57.863922 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:57.863900 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd5ba5d9-23ee-42d4-97f0-3224353b90ec-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:18:57.863922 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:57.863918 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwx2j\" (UniqueName: \"kubernetes.io/projected/16bdf204-496b-4e72-b5cd-e7b5719043e5-kube-api-access-vwx2j\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:18:57.865523 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:57.865502 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd5ba5d9-23ee-42d4-97f0-3224353b90ec-kube-api-access-7zjbx" pod "fd5ba5d9-23ee-42d4-97f0-3224353b90ec" (UID: "fd5ba5d9-23ee-42d4-97f0-3224353b90ec"). InnerVolumeSpecName "kube-api-access-7zjbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:18:57.964872 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:57.964839 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zjbx\" (UniqueName: \"kubernetes.io/projected/fd5ba5d9-23ee-42d4-97f0-3224353b90ec-kube-api-access-7zjbx\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:18:58.110366 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:58.110333 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2e618f-f475b8c6c-lr97t"] Oct 30 02:18:58.114416 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:58.114350 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-2e618f-f475b8c6c-lr97t"] Oct 30 02:18:58.548351 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:58.548320 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16bdf204-496b-4e72-b5cd-e7b5719043e5" path="/var/lib/kubelet/pods/16bdf204-496b-4e72-b5cd-e7b5719043e5/volumes" Oct 30 02:18:58.548945 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:18:58.548919 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd5ba5d9-23ee-42d4-97f0-3224353b90ec" path="/var/lib/kubelet/pods/fd5ba5d9-23ee-42d4-97f0-3224353b90ec/volumes" Oct 30 02:19:00.608429 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:00.608395 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-396f8e-c744b6894-4f4lh"] Oct 30 02:19:00.638227 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:00.638194 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-396f8e-88448749f-cwwq8"] Oct 30 02:19:02.305975 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:02.305942 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4f7628-7fd5f9b886-bgc5n"] Oct 30 02:19:02.337898 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:02.337865 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4f7628-5d4595b5c5-q9f88"] Oct 30 02:19:05.609151 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:05.609101 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-396f8e-c744b6894-4f4lh" podUID="30b6e290-ede2-4b9f-b65d-64d7e14355c5" containerName="payload-processing" containerID="cri-o://3e505be50663b3c77d307fb99d54c4f5c4f4de6db876ce3bb465029e03dcfd41" gracePeriod=25 Oct 30 02:19:05.639185 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:05.639137 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-396f8e-88448749f-cwwq8" podUID="b2d4fdd6-a883-4f5c-90bd-6f36b799ffa0" containerName="payload-pre-processing" containerID="cri-o://45cafb0ca65d1e7fc8c0a6163644325373636108311345d2b64ccd12ce8ef91a" gracePeriod=25 Oct 30 02:19:05.732615 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:05.732579 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 02:19:07.203823 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.203786 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f7628-5cc968cb6c-644x7"] Oct 30 02:19:07.204618 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.204585 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f7628-5cc968cb6c-644x7" podUID="d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2" containerName="maas-api" containerID="cri-o://2cfd9a7fdd62ab716bbbc757598308f6ca969e3f01dc7d22b785931c61c7e3e4" gracePeriod=30 Oct 30 02:19:07.306620 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.306576 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4f7628-7fd5f9b886-bgc5n" podUID="a7f33e22-ce25-46bc-8add-b9aa4434e0e3" containerName="payload-processing" containerID="cri-o://2039a470a99f4e5e0dd3385cc285ed722dbcca96a8fdd3576b5e8cf4e33f2416" gracePeriod=25 Oct 30 02:19:07.338411 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.338370 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4f7628-5d4595b5c5-q9f88" podUID="7f16168b-61d0-4a08-94b7-7ea5581e4005" containerName="payload-pre-processing" containerID="cri-o://c7f698cc3c62befb5e20a345d0255ba0e086dbff130aa1bf2f01bb13825a9328" gracePeriod=25 Oct 30 02:19:07.507974 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.507944 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-396f8e-59798df9ff-qhchc"] Oct 30 02:19:07.508171 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.508148 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-396f8e-59798df9ff-qhchc" podUID="348f8c54-8ced-40e2-846d-f551e78b3092" containerName="maas-api" containerID="cri-o://fe5130e9997b415e33591e0953ba9909def21256a0bf3f5749ddeeeab046ae6b" gracePeriod=30 Oct 30 02:19:07.534522 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.534490 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f7628-5cc968cb6c-644x7" Oct 30 02:19:07.581489 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.580830 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2-kube-api-access-rf488\" (UniqueName: \"kubernetes.io/projected/d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2-kube-api-access-rf488\") pod \"d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2\" (UID: \"d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2\") " Oct 30 02:19:07.581489 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.580909 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2-maas-api-tls\") pod \"d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2\" (UID: \"d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2\") " Oct 30 02:19:07.581489 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.580966 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2-metrics-certs\") pod \"d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2\" (UID: \"d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2\") " Oct 30 02:19:07.581489 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.581032 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2-openshift-service-ca\") pod \"d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2\" (UID: \"d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2\") " Oct 30 02:19:07.581937 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.581893 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2-openshift-service-ca" pod "d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2" (UID: "d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:19:07.583779 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.583749 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2-maas-api-tls" pod "d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2" (UID: "d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:19:07.583992 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.583969 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2-kube-api-access-rf488" pod "d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2" (UID: "d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2"). InnerVolumeSpecName "kube-api-access-rf488". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:07.584973 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.584953 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2-metrics-certs" pod "d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2" (UID: "d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:19:07.682596 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.682563 6221 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2-maas-api-tls\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:07.682596 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.682597 6221 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2-metrics-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:07.682792 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.682611 6221 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2-openshift-service-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:07.682792 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.682627 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf488\" (UniqueName: \"kubernetes.io/projected/d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2-kube-api-access-rf488\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:07.692444 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.692415 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4f7628-5d4595b5c5-q9f88" Oct 30 02:19:07.696084 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.696061 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4f7628-7fd5f9b886-bgc5n" Oct 30 02:19:07.776944 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.776918 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-396f8e-59798df9ff-qhchc" Oct 30 02:19:07.783899 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.783872 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f16168b-61d0-4a08-94b7-7ea5581e4005-kube-api-access-nbl4r\" (UniqueName: \"kubernetes.io/projected/7f16168b-61d0-4a08-94b7-7ea5581e4005-kube-api-access-nbl4r\") pod \"7f16168b-61d0-4a08-94b7-7ea5581e4005\" (UID: \"7f16168b-61d0-4a08-94b7-7ea5581e4005\") " Oct 30 02:19:07.784055 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.784019 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7f33e22-ce25-46bc-8add-b9aa4434e0e3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7f33e22-ce25-46bc-8add-b9aa4434e0e3-plugins-config-volume\") pod \"a7f33e22-ce25-46bc-8add-b9aa4434e0e3\" (UID: \"a7f33e22-ce25-46bc-8add-b9aa4434e0e3\") " Oct 30 02:19:07.784163 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.784141 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7f33e22-ce25-46bc-8add-b9aa4434e0e3-kube-api-access-7ks9s\" (UniqueName: \"kubernetes.io/projected/a7f33e22-ce25-46bc-8add-b9aa4434e0e3-kube-api-access-7ks9s\") pod \"a7f33e22-ce25-46bc-8add-b9aa4434e0e3\" (UID: \"a7f33e22-ce25-46bc-8add-b9aa4434e0e3\") " Oct 30 02:19:07.784311 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.784277 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f16168b-61d0-4a08-94b7-7ea5581e4005-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f16168b-61d0-4a08-94b7-7ea5581e4005-plugins-config-volume\") pod \"7f16168b-61d0-4a08-94b7-7ea5581e4005\" (UID: \"7f16168b-61d0-4a08-94b7-7ea5581e4005\") " Oct 30 02:19:07.784437 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.784393 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7f33e22-ce25-46bc-8add-b9aa4434e0e3-plugins-config-volume" pod "a7f33e22-ce25-46bc-8add-b9aa4434e0e3" (UID: "a7f33e22-ce25-46bc-8add-b9aa4434e0e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:19:07.784922 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.784896 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f16168b-61d0-4a08-94b7-7ea5581e4005-plugins-config-volume" pod "7f16168b-61d0-4a08-94b7-7ea5581e4005" (UID: "7f16168b-61d0-4a08-94b7-7ea5581e4005"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:19:07.785147 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.785127 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7f33e22-ce25-46bc-8add-b9aa4434e0e3-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:07.785229 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.785152 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f16168b-61d0-4a08-94b7-7ea5581e4005-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:07.786671 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.786621 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f16168b-61d0-4a08-94b7-7ea5581e4005-kube-api-access-nbl4r" pod "7f16168b-61d0-4a08-94b7-7ea5581e4005" (UID: "7f16168b-61d0-4a08-94b7-7ea5581e4005"). InnerVolumeSpecName "kube-api-access-nbl4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:07.786899 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.786873 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7f33e22-ce25-46bc-8add-b9aa4434e0e3-kube-api-access-7ks9s" pod "a7f33e22-ce25-46bc-8add-b9aa4434e0e3" (UID: "a7f33e22-ce25-46bc-8add-b9aa4434e0e3"). InnerVolumeSpecName "kube-api-access-7ks9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:07.826959 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.826799 6221 generic.go:356] "Generic (PLEG): container finished" podID="348f8c54-8ced-40e2-846d-f551e78b3092" containerID="fe5130e9997b415e33591e0953ba9909def21256a0bf3f5749ddeeeab046ae6b" exitCode=0 Oct 30 02:19:07.826959 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.826887 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-396f8e-59798df9ff-qhchc" event={"ID":"348f8c54-8ced-40e2-846d-f551e78b3092","Type":"ContainerDied","Data":"fe5130e9997b415e33591e0953ba9909def21256a0bf3f5749ddeeeab046ae6b"} Oct 30 02:19:07.827381 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.827192 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-396f8e-59798df9ff-qhchc" event={"ID":"348f8c54-8ced-40e2-846d-f551e78b3092","Type":"ContainerDied","Data":"b0a53cdc3c0788ce832ce0b4944557f4500d624cf6e11e1497f15dcf4a4756a7"} Oct 30 02:19:07.827381 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.827254 6221 scope.go:122] "RemoveContainer" containerID="fe5130e9997b415e33591e0953ba9909def21256a0bf3f5749ddeeeab046ae6b" Oct 30 02:19:07.827563 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.827417 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-396f8e-59798df9ff-qhchc" Oct 30 02:19:07.830935 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.830906 6221 generic.go:356] "Generic (PLEG): container finished" podID="7f16168b-61d0-4a08-94b7-7ea5581e4005" containerID="c7f698cc3c62befb5e20a345d0255ba0e086dbff130aa1bf2f01bb13825a9328" exitCode=0 Oct 30 02:19:07.831074 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.830998 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4f7628-5d4595b5c5-q9f88" Oct 30 02:19:07.831138 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.830995 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4f7628-5d4595b5c5-q9f88" event={"ID":"7f16168b-61d0-4a08-94b7-7ea5581e4005","Type":"ContainerDied","Data":"c7f698cc3c62befb5e20a345d0255ba0e086dbff130aa1bf2f01bb13825a9328"} Oct 30 02:19:07.831192 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.831130 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4f7628-5d4595b5c5-q9f88" event={"ID":"7f16168b-61d0-4a08-94b7-7ea5581e4005","Type":"ContainerDied","Data":"f071e818f3b496b0ce6baf692330f16f71e76d2fa948a0f2628a73ae90ff5488"} Oct 30 02:19:07.832624 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.832597 6221 generic.go:356] "Generic (PLEG): container finished" podID="a7f33e22-ce25-46bc-8add-b9aa4434e0e3" containerID="2039a470a99f4e5e0dd3385cc285ed722dbcca96a8fdd3576b5e8cf4e33f2416" exitCode=0 Oct 30 02:19:07.832725 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.832687 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4f7628-7fd5f9b886-bgc5n" event={"ID":"a7f33e22-ce25-46bc-8add-b9aa4434e0e3","Type":"ContainerDied","Data":"2039a470a99f4e5e0dd3385cc285ed722dbcca96a8fdd3576b5e8cf4e33f2416"} Oct 30 02:19:07.832771 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.832725 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4f7628-7fd5f9b886-bgc5n" event={"ID":"a7f33e22-ce25-46bc-8add-b9aa4434e0e3","Type":"ContainerDied","Data":"ccf66000ba3884eda091d38f00f4bfcc1b96eccb662fec5230be5f04ee6147f2"} Oct 30 02:19:07.832809 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.832796 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4f7628-7fd5f9b886-bgc5n" Oct 30 02:19:07.839588 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.839514 6221 generic.go:356] "Generic (PLEG): container finished" podID="d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2" containerID="2cfd9a7fdd62ab716bbbc757598308f6ca969e3f01dc7d22b785931c61c7e3e4" exitCode=0 Oct 30 02:19:07.839713 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.839606 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f7628-5cc968cb6c-644x7" event={"ID":"d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2","Type":"ContainerDied","Data":"2cfd9a7fdd62ab716bbbc757598308f6ca969e3f01dc7d22b785931c61c7e3e4"} Oct 30 02:19:07.839713 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.839644 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f7628-5cc968cb6c-644x7" event={"ID":"d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2","Type":"ContainerDied","Data":"89be3ed96d6f87708cf580c88afb48cfbc7383a5c53c862dad2568d21d7ed426"} Oct 30 02:19:07.839847 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.839715 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f7628-5cc968cb6c-644x7" Oct 30 02:19:07.844716 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.844696 6221 scope.go:122] "RemoveContainer" containerID="fe5130e9997b415e33591e0953ba9909def21256a0bf3f5749ddeeeab046ae6b" Oct 30 02:19:07.845144 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:19:07.845121 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe5130e9997b415e33591e0953ba9909def21256a0bf3f5749ddeeeab046ae6b\": container with ID starting with fe5130e9997b415e33591e0953ba9909def21256a0bf3f5749ddeeeab046ae6b not found: ID does not exist" containerID="fe5130e9997b415e33591e0953ba9909def21256a0bf3f5749ddeeeab046ae6b" Oct 30 02:19:07.845251 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.845154 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe5130e9997b415e33591e0953ba9909def21256a0bf3f5749ddeeeab046ae6b"} err="failed to get container status \"fe5130e9997b415e33591e0953ba9909def21256a0bf3f5749ddeeeab046ae6b\": rpc error: code = NotFound desc = could not find container \"fe5130e9997b415e33591e0953ba9909def21256a0bf3f5749ddeeeab046ae6b\": container with ID starting with fe5130e9997b415e33591e0953ba9909def21256a0bf3f5749ddeeeab046ae6b not found: ID does not exist" Oct 30 02:19:07.845251 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.845177 6221 scope.go:122] "RemoveContainer" containerID="c7f698cc3c62befb5e20a345d0255ba0e086dbff130aa1bf2f01bb13825a9328" Oct 30 02:19:07.856102 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.856084 6221 scope.go:122] "RemoveContainer" containerID="c7f698cc3c62befb5e20a345d0255ba0e086dbff130aa1bf2f01bb13825a9328" Oct 30 02:19:07.856375 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:19:07.856353 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7f698cc3c62befb5e20a345d0255ba0e086dbff130aa1bf2f01bb13825a9328\": container with ID starting with c7f698cc3c62befb5e20a345d0255ba0e086dbff130aa1bf2f01bb13825a9328 not found: ID does not exist" containerID="c7f698cc3c62befb5e20a345d0255ba0e086dbff130aa1bf2f01bb13825a9328" Oct 30 02:19:07.856455 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.856386 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7f698cc3c62befb5e20a345d0255ba0e086dbff130aa1bf2f01bb13825a9328"} err="failed to get container status \"c7f698cc3c62befb5e20a345d0255ba0e086dbff130aa1bf2f01bb13825a9328\": rpc error: code = NotFound desc = could not find container \"c7f698cc3c62befb5e20a345d0255ba0e086dbff130aa1bf2f01bb13825a9328\": container with ID starting with c7f698cc3c62befb5e20a345d0255ba0e086dbff130aa1bf2f01bb13825a9328 not found: ID does not exist" Oct 30 02:19:07.856455 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.856413 6221 scope.go:122] "RemoveContainer" containerID="2039a470a99f4e5e0dd3385cc285ed722dbcca96a8fdd3576b5e8cf4e33f2416" Oct 30 02:19:07.866275 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.866254 6221 scope.go:122] "RemoveContainer" containerID="2039a470a99f4e5e0dd3385cc285ed722dbcca96a8fdd3576b5e8cf4e33f2416" Oct 30 02:19:07.866562 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:19:07.866517 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2039a470a99f4e5e0dd3385cc285ed722dbcca96a8fdd3576b5e8cf4e33f2416\": container with ID starting with 2039a470a99f4e5e0dd3385cc285ed722dbcca96a8fdd3576b5e8cf4e33f2416 not found: ID does not exist" containerID="2039a470a99f4e5e0dd3385cc285ed722dbcca96a8fdd3576b5e8cf4e33f2416" Oct 30 02:19:07.866562 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.866556 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2039a470a99f4e5e0dd3385cc285ed722dbcca96a8fdd3576b5e8cf4e33f2416"} err="failed to get container status \"2039a470a99f4e5e0dd3385cc285ed722dbcca96a8fdd3576b5e8cf4e33f2416\": rpc error: code = NotFound desc = could not find container \"2039a470a99f4e5e0dd3385cc285ed722dbcca96a8fdd3576b5e8cf4e33f2416\": container with ID starting with 2039a470a99f4e5e0dd3385cc285ed722dbcca96a8fdd3576b5e8cf4e33f2416 not found: ID does not exist" Oct 30 02:19:07.866716 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.866573 6221 scope.go:122] "RemoveContainer" containerID="2cfd9a7fdd62ab716bbbc757598308f6ca969e3f01dc7d22b785931c61c7e3e4" Oct 30 02:19:07.875498 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.875478 6221 scope.go:122] "RemoveContainer" containerID="2cfd9a7fdd62ab716bbbc757598308f6ca969e3f01dc7d22b785931c61c7e3e4" Oct 30 02:19:07.875823 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:19:07.875800 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cfd9a7fdd62ab716bbbc757598308f6ca969e3f01dc7d22b785931c61c7e3e4\": container with ID starting with 2cfd9a7fdd62ab716bbbc757598308f6ca969e3f01dc7d22b785931c61c7e3e4 not found: ID does not exist" containerID="2cfd9a7fdd62ab716bbbc757598308f6ca969e3f01dc7d22b785931c61c7e3e4" Oct 30 02:19:07.875915 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.875828 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cfd9a7fdd62ab716bbbc757598308f6ca969e3f01dc7d22b785931c61c7e3e4"} err="failed to get container status \"2cfd9a7fdd62ab716bbbc757598308f6ca969e3f01dc7d22b785931c61c7e3e4\": rpc error: code = NotFound desc = could not find container \"2cfd9a7fdd62ab716bbbc757598308f6ca969e3f01dc7d22b785931c61c7e3e4\": container with ID starting with 2cfd9a7fdd62ab716bbbc757598308f6ca969e3f01dc7d22b785931c61c7e3e4 not found: ID does not exist" Oct 30 02:19:07.886302 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.886279 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/348f8c54-8ced-40e2-846d-f551e78b3092-kube-api-access-8b774\" (UniqueName: \"kubernetes.io/projected/348f8c54-8ced-40e2-846d-f551e78b3092-kube-api-access-8b774\") pod \"348f8c54-8ced-40e2-846d-f551e78b3092\" (UID: \"348f8c54-8ced-40e2-846d-f551e78b3092\") " Oct 30 02:19:07.886434 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.886325 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/348f8c54-8ced-40e2-846d-f551e78b3092-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/348f8c54-8ced-40e2-846d-f551e78b3092-openshift-service-ca\") pod \"348f8c54-8ced-40e2-846d-f551e78b3092\" (UID: \"348f8c54-8ced-40e2-846d-f551e78b3092\") " Oct 30 02:19:07.886512 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.886445 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/348f8c54-8ced-40e2-846d-f551e78b3092-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/348f8c54-8ced-40e2-846d-f551e78b3092-maas-api-tls\") pod \"348f8c54-8ced-40e2-846d-f551e78b3092\" (UID: \"348f8c54-8ced-40e2-846d-f551e78b3092\") " Oct 30 02:19:07.886512 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.886463 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/348f8c54-8ced-40e2-846d-f551e78b3092-metrics-certs\" (UniqueName: \"kubernetes.io/secret/348f8c54-8ced-40e2-846d-f551e78b3092-metrics-certs\") pod \"348f8c54-8ced-40e2-846d-f551e78b3092\" (UID: \"348f8c54-8ced-40e2-846d-f551e78b3092\") " Oct 30 02:19:07.886708 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.886686 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/348f8c54-8ced-40e2-846d-f551e78b3092-openshift-service-ca" pod "348f8c54-8ced-40e2-846d-f551e78b3092" (UID: "348f8c54-8ced-40e2-846d-f551e78b3092"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:19:07.886936 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.886917 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbl4r\" (UniqueName: \"kubernetes.io/projected/7f16168b-61d0-4a08-94b7-7ea5581e4005-kube-api-access-nbl4r\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:07.886991 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.886943 6221 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/348f8c54-8ced-40e2-846d-f551e78b3092-openshift-service-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:07.886991 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.886958 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ks9s\" (UniqueName: \"kubernetes.io/projected/a7f33e22-ce25-46bc-8add-b9aa4434e0e3-kube-api-access-7ks9s\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:07.888520 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.888494 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/348f8c54-8ced-40e2-846d-f551e78b3092-kube-api-access-8b774" pod "348f8c54-8ced-40e2-846d-f551e78b3092" (UID: "348f8c54-8ced-40e2-846d-f551e78b3092"). InnerVolumeSpecName "kube-api-access-8b774". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:07.888662 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.888519 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/348f8c54-8ced-40e2-846d-f551e78b3092-maas-api-tls" pod "348f8c54-8ced-40e2-846d-f551e78b3092" (UID: "348f8c54-8ced-40e2-846d-f551e78b3092"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:19:07.888662 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.888563 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/348f8c54-8ced-40e2-846d-f551e78b3092-metrics-certs" pod "348f8c54-8ced-40e2-846d-f551e78b3092" (UID: "348f8c54-8ced-40e2-846d-f551e78b3092"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:19:07.927456 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.927426 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f7628-5cc968cb6c-644x7"] Oct 30 02:19:07.931480 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.931458 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4f7628-5cc968cb6c-644x7"] Oct 30 02:19:07.984496 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.984445 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4f7628-5d4595b5c5-q9f88"] Oct 30 02:19:07.984778 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.984512 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4f7628-5d4595b5c5-q9f88"] Oct 30 02:19:07.988686 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.988654 6221 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/348f8c54-8ced-40e2-846d-f551e78b3092-maas-api-tls\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:07.988686 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.988687 6221 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/348f8c54-8ced-40e2-846d-f551e78b3092-metrics-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:07.988951 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:07.988701 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b774\" (UniqueName: \"kubernetes.io/projected/348f8c54-8ced-40e2-846d-f551e78b3092-kube-api-access-8b774\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:08.009564 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:08.009510 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4f7628-7fd5f9b886-bgc5n"] Oct 30 02:19:08.012946 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:08.012922 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4f7628-7fd5f9b886-bgc5n"] Oct 30 02:19:08.173721 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:08.173685 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-396f8e-59798df9ff-qhchc"] Oct 30 02:19:08.184788 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:08.184759 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-396f8e-59798df9ff-qhchc"] Oct 30 02:19:08.548779 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:08.548743 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="348f8c54-8ced-40e2-846d-f551e78b3092" path="/var/lib/kubelet/pods/348f8c54-8ced-40e2-846d-f551e78b3092/volumes" Oct 30 02:19:08.549326 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:08.549303 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f16168b-61d0-4a08-94b7-7ea5581e4005" path="/var/lib/kubelet/pods/7f16168b-61d0-4a08-94b7-7ea5581e4005/volumes" Oct 30 02:19:08.549758 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:08.549740 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7f33e22-ce25-46bc-8add-b9aa4434e0e3" path="/var/lib/kubelet/pods/a7f33e22-ce25-46bc-8add-b9aa4434e0e3/volumes" Oct 30 02:19:08.550609 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:08.550591 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2" path="/var/lib/kubelet/pods/d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2/volumes" Oct 30 02:19:11.127765 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:11.127724 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-396f8e-c744b6894-4f4lh" podUID="30b6e290-ede2-4b9f-b65d-64d7e14355c5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:19:11.128261 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:11.127792 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-396f8e-88448749f-cwwq8" podUID="b2d4fdd6-a883-4f5c-90bd-6f36b799ffa0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:19:20.062235 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:20.062193 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92"] Oct 30 02:19:20.083127 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:20.083090 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz"] Oct 30 02:19:21.127218 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:21.127170 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-396f8e-88448749f-cwwq8" podUID="b2d4fdd6-a883-4f5c-90bd-6f36b799ffa0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:19:21.127648 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:21.127215 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-396f8e-c744b6894-4f4lh" podUID="30b6e290-ede2-4b9f-b65d-64d7e14355c5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:19:21.494418 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:19:21.494328 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 02:19:25.711208 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:25.711175 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-558327-bd6c69d5b-scg4b"] Oct 30 02:19:25.942085 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:25.941580 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-396f8e-c744b6894-4f4lh" Oct 30 02:19:25.957866 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:25.957827 6221 generic.go:356] "Generic (PLEG): container finished" podID="b2d4fdd6-a883-4f5c-90bd-6f36b799ffa0" containerID="45cafb0ca65d1e7fc8c0a6163644325373636108311345d2b64ccd12ce8ef91a" exitCode=0 Oct 30 02:19:25.958049 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:25.957961 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-396f8e-88448749f-cwwq8" event={"ID":"b2d4fdd6-a883-4f5c-90bd-6f36b799ffa0","Type":"ContainerDied","Data":"45cafb0ca65d1e7fc8c0a6163644325373636108311345d2b64ccd12ce8ef91a"} Oct 30 02:19:25.958049 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:25.958013 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-396f8e-88448749f-cwwq8" event={"ID":"b2d4fdd6-a883-4f5c-90bd-6f36b799ffa0","Type":"ContainerDied","Data":"50b54b313503ad56e21689a869b8e8a592aa5df43eca7b0921de56a708624758"} Oct 30 02:19:25.958049 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:25.958035 6221 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50b54b313503ad56e21689a869b8e8a592aa5df43eca7b0921de56a708624758" Oct 30 02:19:25.961296 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:25.960930 6221 generic.go:356] "Generic (PLEG): container finished" podID="30b6e290-ede2-4b9f-b65d-64d7e14355c5" containerID="3e505be50663b3c77d307fb99d54c4f5c4f4de6db876ce3bb465029e03dcfd41" exitCode=0 Oct 30 02:19:25.961296 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:25.960986 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-396f8e-c744b6894-4f4lh" event={"ID":"30b6e290-ede2-4b9f-b65d-64d7e14355c5","Type":"ContainerDied","Data":"3e505be50663b3c77d307fb99d54c4f5c4f4de6db876ce3bb465029e03dcfd41"} Oct 30 02:19:25.961296 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:25.961014 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-396f8e-c744b6894-4f4lh" Oct 30 02:19:25.961296 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:25.961035 6221 scope.go:122] "RemoveContainer" containerID="3e505be50663b3c77d307fb99d54c4f5c4f4de6db876ce3bb465029e03dcfd41" Oct 30 02:19:25.961296 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:25.961024 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-396f8e-c744b6894-4f4lh" event={"ID":"30b6e290-ede2-4b9f-b65d-64d7e14355c5","Type":"ContainerDied","Data":"6168b948a1927f2d7484a04d9108cf14a7277eff0dd81594698dd385fc75ef45"} Oct 30 02:19:25.971047 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:25.971018 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-396f8e-88448749f-cwwq8" Oct 30 02:19:25.973244 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:25.973225 6221 scope.go:122] "RemoveContainer" containerID="3e505be50663b3c77d307fb99d54c4f5c4f4de6db876ce3bb465029e03dcfd41" Oct 30 02:19:25.973515 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:19:25.973495 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e505be50663b3c77d307fb99d54c4f5c4f4de6db876ce3bb465029e03dcfd41\": container with ID starting with 3e505be50663b3c77d307fb99d54c4f5c4f4de6db876ce3bb465029e03dcfd41 not found: ID does not exist" containerID="3e505be50663b3c77d307fb99d54c4f5c4f4de6db876ce3bb465029e03dcfd41" Oct 30 02:19:25.973620 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:25.973524 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e505be50663b3c77d307fb99d54c4f5c4f4de6db876ce3bb465029e03dcfd41"} err="failed to get container status \"3e505be50663b3c77d307fb99d54c4f5c4f4de6db876ce3bb465029e03dcfd41\": rpc error: code = NotFound desc = could not find container \"3e505be50663b3c77d307fb99d54c4f5c4f4de6db876ce3bb465029e03dcfd41\": container with ID starting with 3e505be50663b3c77d307fb99d54c4f5c4f4de6db876ce3bb465029e03dcfd41 not found: ID does not exist" Oct 30 02:19:26.016896 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:26.016864 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30b6e290-ede2-4b9f-b65d-64d7e14355c5-kube-api-access-kdlnr\" (UniqueName: \"kubernetes.io/projected/30b6e290-ede2-4b9f-b65d-64d7e14355c5-kube-api-access-kdlnr\") pod \"30b6e290-ede2-4b9f-b65d-64d7e14355c5\" (UID: \"30b6e290-ede2-4b9f-b65d-64d7e14355c5\") " Oct 30 02:19:26.017054 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:26.017025 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30b6e290-ede2-4b9f-b65d-64d7e14355c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30b6e290-ede2-4b9f-b65d-64d7e14355c5-plugins-config-volume\") pod \"30b6e290-ede2-4b9f-b65d-64d7e14355c5\" (UID: \"30b6e290-ede2-4b9f-b65d-64d7e14355c5\") " Oct 30 02:19:26.017389 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:26.017368 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30b6e290-ede2-4b9f-b65d-64d7e14355c5-plugins-config-volume" pod "30b6e290-ede2-4b9f-b65d-64d7e14355c5" (UID: "30b6e290-ede2-4b9f-b65d-64d7e14355c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:19:26.017561 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:26.017517 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30b6e290-ede2-4b9f-b65d-64d7e14355c5-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:26.018949 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:26.018929 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30b6e290-ede2-4b9f-b65d-64d7e14355c5-kube-api-access-kdlnr" pod "30b6e290-ede2-4b9f-b65d-64d7e14355c5" (UID: "30b6e290-ede2-4b9f-b65d-64d7e14355c5"). InnerVolumeSpecName "kube-api-access-kdlnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:26.118452 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:26.118383 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2d4fdd6-a883-4f5c-90bd-6f36b799ffa0-kube-api-access-57mjn\" (UniqueName: \"kubernetes.io/projected/b2d4fdd6-a883-4f5c-90bd-6f36b799ffa0-kube-api-access-57mjn\") pod \"b2d4fdd6-a883-4f5c-90bd-6f36b799ffa0\" (UID: \"b2d4fdd6-a883-4f5c-90bd-6f36b799ffa0\") " Oct 30 02:19:26.118619 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:26.118591 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2d4fdd6-a883-4f5c-90bd-6f36b799ffa0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2d4fdd6-a883-4f5c-90bd-6f36b799ffa0-plugins-config-volume\") pod \"b2d4fdd6-a883-4f5c-90bd-6f36b799ffa0\" (UID: \"b2d4fdd6-a883-4f5c-90bd-6f36b799ffa0\") " Oct 30 02:19:26.118987 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:26.118961 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdlnr\" (UniqueName: \"kubernetes.io/projected/30b6e290-ede2-4b9f-b65d-64d7e14355c5-kube-api-access-kdlnr\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:26.118987 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:26.118981 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2d4fdd6-a883-4f5c-90bd-6f36b799ffa0-plugins-config-volume" pod "b2d4fdd6-a883-4f5c-90bd-6f36b799ffa0" (UID: "b2d4fdd6-a883-4f5c-90bd-6f36b799ffa0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:19:26.120448 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:26.120427 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2d4fdd6-a883-4f5c-90bd-6f36b799ffa0-kube-api-access-57mjn" pod "b2d4fdd6-a883-4f5c-90bd-6f36b799ffa0" (UID: "b2d4fdd6-a883-4f5c-90bd-6f36b799ffa0"). InnerVolumeSpecName "kube-api-access-57mjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:26.220072 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:26.220040 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2d4fdd6-a883-4f5c-90bd-6f36b799ffa0-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:26.220072 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:26.220068 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57mjn\" (UniqueName: \"kubernetes.io/projected/b2d4fdd6-a883-4f5c-90bd-6f36b799ffa0-kube-api-access-57mjn\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:26.326047 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:26.326004 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-396f8e-c744b6894-4f4lh"] Oct 30 02:19:26.331337 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:26.330257 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-396f8e-c744b6894-4f4lh"] Oct 30 02:19:26.432185 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:26.432104 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 02:19:26.452491 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:26.437647 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c8d376-76c47844b8-q84n6"] Oct 30 02:19:26.465222 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:26.465188 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d376-f998788b7-gvrjq"] Oct 30 02:19:26.548568 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:26.548519 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30b6e290-ede2-4b9f-b65d-64d7e14355c5" path="/var/lib/kubelet/pods/30b6e290-ede2-4b9f-b65d-64d7e14355c5/volumes" Oct 30 02:19:26.968878 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:26.968848 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-396f8e-88448749f-cwwq8" Oct 30 02:19:27.024861 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:27.024817 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-396f8e-88448749f-cwwq8"] Oct 30 02:19:27.039423 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:27.039388 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-396f8e-88448749f-cwwq8"] Oct 30 02:19:28.547270 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:28.547235 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2d4fdd6-a883-4f5c-90bd-6f36b799ffa0" path="/var/lib/kubelet/pods/b2d4fdd6-a883-4f5c-90bd-6f36b799ffa0/volumes" Oct 30 02:19:30.050217 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:30.050181 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-340ae6-8f55557c5-c68h9"] Oct 30 02:19:30.712431 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:30.712384 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-558327-bd6c69d5b-scg4b" podUID="125fc809-a755-460c-8605-363489a85b49" containerName="payload-processing" containerID="cri-o://b8652bab3d6525ebae066d2815fac5993d0c3e10a7347e93eeda9a901338de46" gracePeriod=25 Oct 30 02:19:31.438338 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:31.438300 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d376-76c47844b8-q84n6" podUID="6c21d89b-1302-47e9-a329-ff6e4bd5e10e" containerName="payload-processing" containerID="cri-o://a40c22e280ac14b47a74a9c49a6a0a8438e1b46570620095299577cca1f2e3ae" gracePeriod=25 Oct 30 02:19:31.465507 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:31.465424 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d376-f998788b7-gvrjq" podUID="28dc66b5-3755-4151-a2a3-4dbc793d0c28" containerName="payload-pre-processing" containerID="cri-o://1e65a93666462732ba6be5470aa969cfdb5fad1a3ec057662f646c46e6680ee6" gracePeriod=25 Oct 30 02:19:31.900092 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:31.900045 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d376-76c47844b8-q84n6" podUID="6c21d89b-1302-47e9-a329-ff6e4bd5e10e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:19:31.900092 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:31.900079 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d376-f998788b7-gvrjq" podUID="28dc66b5-3755-4151-a2a3-4dbc793d0c28" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:19:35.051527 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:35.051455 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-340ae6-8f55557c5-c68h9" podUID="cc95feca-80e8-41ac-8a29-be0b9c840d1d" containerName="payload-pre-processing" containerID="cri-o://6062f35df1eabd36ad3cf281db91d864e1c1cc759bf12b732fde5a74cc85ac07" gracePeriod=25 Oct 30 02:19:35.076309 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:35.076266 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-558327-bd6c69d5b-scg4b" podUID="125fc809-a755-460c-8605-363489a85b49" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:19:35.087921 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:35.087870 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" podUID="ea9583aa-8259-4c9b-932e-2d6559008d86" containerName="main" containerID="cri-o://695ecc7193684eab12d84f19975b89e440ddee0391e3d1dd630856d4ace3125d" gracePeriod=45 Oct 30 02:19:35.103628 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:35.103583 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" podUID="354ec282-b972-4fca-b4e3-1e33ef461f2f" containerName="main" containerID="cri-o://65caa84f2e160303cc7c87a7886dac96fe8738d0a8dd595b9c4c2372d9100601" gracePeriod=45 Oct 30 02:19:35.368877 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:35.368844 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" Oct 30 02:19:35.397932 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:35.397905 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" Oct 30 02:19:35.427615 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:35.427582 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea9583aa-8259-4c9b-932e-2d6559008d86-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea9583aa-8259-4c9b-932e-2d6559008d86-dshm\") pod \"ea9583aa-8259-4c9b-932e-2d6559008d86\" (UID: \"ea9583aa-8259-4c9b-932e-2d6559008d86\") " Oct 30 02:19:35.427809 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:35.427627 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea9583aa-8259-4c9b-932e-2d6559008d86-home\" (UniqueName: \"kubernetes.io/empty-dir/ea9583aa-8259-4c9b-932e-2d6559008d86-home\") pod \"ea9583aa-8259-4c9b-932e-2d6559008d86\" (UID: \"ea9583aa-8259-4c9b-932e-2d6559008d86\") " Oct 30 02:19:35.427809 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:35.427689 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/354ec282-b972-4fca-b4e3-1e33ef461f2f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/354ec282-b972-4fca-b4e3-1e33ef461f2f-tmp-dir\") pod \"354ec282-b972-4fca-b4e3-1e33ef461f2f\" (UID: \"354ec282-b972-4fca-b4e3-1e33ef461f2f\") " Oct 30 02:19:35.427809 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:35.427718 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea9583aa-8259-4c9b-932e-2d6559008d86-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea9583aa-8259-4c9b-932e-2d6559008d86-tmp-dir\") pod \"ea9583aa-8259-4c9b-932e-2d6559008d86\" (UID: \"ea9583aa-8259-4c9b-932e-2d6559008d86\") " Oct 30 02:19:35.427809 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:35.427775 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/354ec282-b972-4fca-b4e3-1e33ef461f2f-tls-certs\" (UniqueName: \"kubernetes.io/secret/354ec282-b972-4fca-b4e3-1e33ef461f2f-tls-certs\") pod \"354ec282-b972-4fca-b4e3-1e33ef461f2f\" (UID: \"354ec282-b972-4fca-b4e3-1e33ef461f2f\") " Oct 30 02:19:35.427809 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:35.427799 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/354ec282-b972-4fca-b4e3-1e33ef461f2f-home\" (UniqueName: \"kubernetes.io/empty-dir/354ec282-b972-4fca-b4e3-1e33ef461f2f-home\") pod \"354ec282-b972-4fca-b4e3-1e33ef461f2f\" (UID: \"354ec282-b972-4fca-b4e3-1e33ef461f2f\") " Oct 30 02:19:35.428118 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:35.427859 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea9583aa-8259-4c9b-932e-2d6559008d86-tls-certs\" (UniqueName: \"kubernetes.io/secret/ea9583aa-8259-4c9b-932e-2d6559008d86-tls-certs\") pod \"ea9583aa-8259-4c9b-932e-2d6559008d86\" (UID: \"ea9583aa-8259-4c9b-932e-2d6559008d86\") " Oct 30 02:19:35.428118 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:35.427911 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/354ec282-b972-4fca-b4e3-1e33ef461f2f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/354ec282-b972-4fca-b4e3-1e33ef461f2f-model-cache\") pod \"354ec282-b972-4fca-b4e3-1e33ef461f2f\" (UID: \"354ec282-b972-4fca-b4e3-1e33ef461f2f\") " Oct 30 02:19:35.428118 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:35.427976 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea9583aa-8259-4c9b-932e-2d6559008d86-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea9583aa-8259-4c9b-932e-2d6559008d86-model-cache\") pod \"ea9583aa-8259-4c9b-932e-2d6559008d86\" (UID: \"ea9583aa-8259-4c9b-932e-2d6559008d86\") " Oct 30 02:19:35.428118 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:35.428036 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/354ec282-b972-4fca-b4e3-1e33ef461f2f-kube-api-access-7qgmt\" (UniqueName: \"kubernetes.io/projected/354ec282-b972-4fca-b4e3-1e33ef461f2f-kube-api-access-7qgmt\") pod \"354ec282-b972-4fca-b4e3-1e33ef461f2f\" (UID: \"354ec282-b972-4fca-b4e3-1e33ef461f2f\") " Oct 30 02:19:35.428118 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:35.428093 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea9583aa-8259-4c9b-932e-2d6559008d86-kube-api-access-lhwmq\" (UniqueName: \"kubernetes.io/projected/ea9583aa-8259-4c9b-932e-2d6559008d86-kube-api-access-lhwmq\") pod \"ea9583aa-8259-4c9b-932e-2d6559008d86\" (UID: \"ea9583aa-8259-4c9b-932e-2d6559008d86\") " Oct 30 02:19:35.428405 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:35.428139 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/354ec282-b972-4fca-b4e3-1e33ef461f2f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/354ec282-b972-4fca-b4e3-1e33ef461f2f-dshm\") pod \"354ec282-b972-4fca-b4e3-1e33ef461f2f\" (UID: \"354ec282-b972-4fca-b4e3-1e33ef461f2f\") " Oct 30 02:19:35.428405 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:35.428198 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea9583aa-8259-4c9b-932e-2d6559008d86-tmp-dir" pod "ea9583aa-8259-4c9b-932e-2d6559008d86" (UID: "ea9583aa-8259-4c9b-932e-2d6559008d86"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:35.428405 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:35.428262 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea9583aa-8259-4c9b-932e-2d6559008d86-home" pod "ea9583aa-8259-4c9b-932e-2d6559008d86" (UID: "ea9583aa-8259-4c9b-932e-2d6559008d86"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:35.428603 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:35.428405 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/354ec282-b972-4fca-b4e3-1e33ef461f2f-model-cache" pod "354ec282-b972-4fca-b4e3-1e33ef461f2f" (UID: "354ec282-b972-4fca-b4e3-1e33ef461f2f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:35.428603 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:35.428465 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/354ec282-b972-4fca-b4e3-1e33ef461f2f-tmp-dir" pod "354ec282-b972-4fca-b4e3-1e33ef461f2f" (UID: "354ec282-b972-4fca-b4e3-1e33ef461f2f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:35.428741 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:35.428713 6221 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea9583aa-8259-4c9b-932e-2d6559008d86-home\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:35.428807 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:35.428770 6221 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/354ec282-b972-4fca-b4e3-1e33ef461f2f-tmp-dir\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:35.428807 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:35.428786 6221 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea9583aa-8259-4c9b-932e-2d6559008d86-tmp-dir\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:35.428807 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:35.428798 6221 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/354ec282-b972-4fca-b4e3-1e33ef461f2f-model-cache\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:35.429048 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:35.429029 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea9583aa-8259-4c9b-932e-2d6559008d86-model-cache" pod "ea9583aa-8259-4c9b-932e-2d6559008d86" (UID: "ea9583aa-8259-4c9b-932e-2d6559008d86"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:35.429456 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:35.429415 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/354ec282-b972-4fca-b4e3-1e33ef461f2f-home" pod "354ec282-b972-4fca-b4e3-1e33ef461f2f" (UID: "354ec282-b972-4fca-b4e3-1e33ef461f2f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:35.434190 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:35.434156 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/354ec282-b972-4fca-b4e3-1e33ef461f2f-tls-certs" pod "354ec282-b972-4fca-b4e3-1e33ef461f2f" (UID: "354ec282-b972-4fca-b4e3-1e33ef461f2f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:19:35.434979 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:35.434884 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea9583aa-8259-4c9b-932e-2d6559008d86-dshm" pod "ea9583aa-8259-4c9b-932e-2d6559008d86" (UID: "ea9583aa-8259-4c9b-932e-2d6559008d86"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:35.434979 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:35.434953 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/354ec282-b972-4fca-b4e3-1e33ef461f2f-dshm" pod "354ec282-b972-4fca-b4e3-1e33ef461f2f" (UID: "354ec282-b972-4fca-b4e3-1e33ef461f2f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:35.434979 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:35.434946 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/354ec282-b972-4fca-b4e3-1e33ef461f2f-kube-api-access-7qgmt" pod "354ec282-b972-4fca-b4e3-1e33ef461f2f" (UID: "354ec282-b972-4fca-b4e3-1e33ef461f2f"). InnerVolumeSpecName "kube-api-access-7qgmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:35.435223 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:35.435150 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea9583aa-8259-4c9b-932e-2d6559008d86-tls-certs" pod "ea9583aa-8259-4c9b-932e-2d6559008d86" (UID: "ea9583aa-8259-4c9b-932e-2d6559008d86"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:19:35.437016 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:35.436988 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea9583aa-8259-4c9b-932e-2d6559008d86-kube-api-access-lhwmq" pod "ea9583aa-8259-4c9b-932e-2d6559008d86" (UID: "ea9583aa-8259-4c9b-932e-2d6559008d86"). InnerVolumeSpecName "kube-api-access-lhwmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:35.530301 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:35.530255 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhwmq\" (UniqueName: \"kubernetes.io/projected/ea9583aa-8259-4c9b-932e-2d6559008d86-kube-api-access-lhwmq\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:35.530301 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:35.530297 6221 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/354ec282-b972-4fca-b4e3-1e33ef461f2f-dshm\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:35.530577 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:35.530311 6221 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea9583aa-8259-4c9b-932e-2d6559008d86-dshm\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:35.530577 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:35.530323 6221 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/354ec282-b972-4fca-b4e3-1e33ef461f2f-tls-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:35.530577 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:35.530332 6221 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/354ec282-b972-4fca-b4e3-1e33ef461f2f-home\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:35.530577 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:35.530343 6221 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea9583aa-8259-4c9b-932e-2d6559008d86-tls-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:35.530577 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:35.530354 6221 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea9583aa-8259-4c9b-932e-2d6559008d86-model-cache\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:35.530577 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:35.530362 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qgmt\" (UniqueName: \"kubernetes.io/projected/354ec282-b972-4fca-b4e3-1e33ef461f2f-kube-api-access-7qgmt\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:36.021980 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:36.021938 6221 generic.go:356] "Generic (PLEG): container finished" podID="ea9583aa-8259-4c9b-932e-2d6559008d86" containerID="695ecc7193684eab12d84f19975b89e440ddee0391e3d1dd630856d4ace3125d" exitCode=0 Oct 30 02:19:36.022205 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:36.021997 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" Oct 30 02:19:36.022205 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:36.022031 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" event={"ID":"ea9583aa-8259-4c9b-932e-2d6559008d86","Type":"ContainerDied","Data":"695ecc7193684eab12d84f19975b89e440ddee0391e3d1dd630856d4ace3125d"} Oct 30 02:19:36.022205 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:36.022072 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" event={"ID":"ea9583aa-8259-4c9b-932e-2d6559008d86","Type":"ContainerDied","Data":"273265d7aef835e93e9ab136ec3542f2a46b4c15ac6416e358a87cf8c6e9241f"} Oct 30 02:19:36.022205 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:36.022096 6221 scope.go:122] "RemoveContainer" containerID="695ecc7193684eab12d84f19975b89e440ddee0391e3d1dd630856d4ace3125d" Oct 30 02:19:36.023579 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:36.023551 6221 generic.go:356] "Generic (PLEG): container finished" podID="354ec282-b972-4fca-b4e3-1e33ef461f2f" containerID="65caa84f2e160303cc7c87a7886dac96fe8738d0a8dd595b9c4c2372d9100601" exitCode=0 Oct 30 02:19:36.023722 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:36.023598 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" Oct 30 02:19:36.023722 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:36.023653 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" event={"ID":"354ec282-b972-4fca-b4e3-1e33ef461f2f","Type":"ContainerDied","Data":"65caa84f2e160303cc7c87a7886dac96fe8738d0a8dd595b9c4c2372d9100601"} Oct 30 02:19:36.023722 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:36.023696 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz" event={"ID":"354ec282-b972-4fca-b4e3-1e33ef461f2f","Type":"ContainerDied","Data":"42e36407ed599fa6ade43a38526b945b4b0a525f30dc3dff65a58ec2f1816308"} Oct 30 02:19:36.032195 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:36.032177 6221 scope.go:122] "RemoveContainer" containerID="695ecc7193684eab12d84f19975b89e440ddee0391e3d1dd630856d4ace3125d" Oct 30 02:19:36.032567 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:19:36.032524 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"695ecc7193684eab12d84f19975b89e440ddee0391e3d1dd630856d4ace3125d\": container with ID starting with 695ecc7193684eab12d84f19975b89e440ddee0391e3d1dd630856d4ace3125d not found: ID does not exist" containerID="695ecc7193684eab12d84f19975b89e440ddee0391e3d1dd630856d4ace3125d" Oct 30 02:19:36.032638 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:36.032577 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"695ecc7193684eab12d84f19975b89e440ddee0391e3d1dd630856d4ace3125d"} err="failed to get container status \"695ecc7193684eab12d84f19975b89e440ddee0391e3d1dd630856d4ace3125d\": rpc error: code = NotFound desc = could not find container \"695ecc7193684eab12d84f19975b89e440ddee0391e3d1dd630856d4ace3125d\": container with ID starting with 695ecc7193684eab12d84f19975b89e440ddee0391e3d1dd630856d4ace3125d not found: ID does not exist" Oct 30 02:19:36.032638 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:36.032595 6221 scope.go:122] "RemoveContainer" containerID="65caa84f2e160303cc7c87a7886dac96fe8738d0a8dd595b9c4c2372d9100601" Oct 30 02:19:36.041823 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:36.041808 6221 scope.go:122] "RemoveContainer" containerID="65caa84f2e160303cc7c87a7886dac96fe8738d0a8dd595b9c4c2372d9100601" Oct 30 02:19:36.042089 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:19:36.042069 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65caa84f2e160303cc7c87a7886dac96fe8738d0a8dd595b9c4c2372d9100601\": container with ID starting with 65caa84f2e160303cc7c87a7886dac96fe8738d0a8dd595b9c4c2372d9100601 not found: ID does not exist" containerID="65caa84f2e160303cc7c87a7886dac96fe8738d0a8dd595b9c4c2372d9100601" Oct 30 02:19:36.042143 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:36.042097 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65caa84f2e160303cc7c87a7886dac96fe8738d0a8dd595b9c4c2372d9100601"} err="failed to get container status \"65caa84f2e160303cc7c87a7886dac96fe8738d0a8dd595b9c4c2372d9100601\": rpc error: code = NotFound desc = could not find container \"65caa84f2e160303cc7c87a7886dac96fe8738d0a8dd595b9c4c2372d9100601\": container with ID starting with 65caa84f2e160303cc7c87a7886dac96fe8738d0a8dd595b9c4c2372d9100601 not found: ID does not exist" Oct 30 02:19:36.069101 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:36.069072 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-340ae6-8f55557c5-c68h9" podUID="cc95feca-80e8-41ac-8a29-be0b9c840d1d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:19:36.074041 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:36.074016 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz"] Oct 30 02:19:36.086630 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:36.086603 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-558327/distinct-w3-558327-kserve-7df6f6f9d6-5vzrz"] Oct 30 02:19:36.119933 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:36.119900 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92"] Oct 30 02:19:36.122945 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:36.122924 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92"] Oct 30 02:19:36.334854 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:36.334810 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-558327/distinct-2-w3-558327-kserve-6c6bb7cdb5-ghc92" podUID="ea9583aa-8259-4c9b-932e-2d6559008d86" containerName="main" probeResult="failure" output="Get \"https://10.134.0.62:8000/ready\": context deadline exceeded" Oct 30 02:19:36.548544 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:36.548508 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="354ec282-b972-4fca-b4e3-1e33ef461f2f" path="/var/lib/kubelet/pods/354ec282-b972-4fca-b4e3-1e33ef461f2f/volumes" Oct 30 02:19:36.548966 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:36.548951 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea9583aa-8259-4c9b-932e-2d6559008d86" path="/var/lib/kubelet/pods/ea9583aa-8259-4c9b-932e-2d6559008d86/volumes" Oct 30 02:19:41.900065 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:41.899969 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d376-f998788b7-gvrjq" podUID="28dc66b5-3755-4151-a2a3-4dbc793d0c28" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:19:41.900524 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:41.899984 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d376-76c47844b8-q84n6" podUID="6c21d89b-1302-47e9-a329-ff6e4bd5e10e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:19:43.776250 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:43.776216 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl"] Oct 30 02:19:45.076575 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:45.076514 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-558327-bd6c69d5b-scg4b" podUID="125fc809-a755-460c-8605-363489a85b49" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:19:46.068759 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:46.068720 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-340ae6-8f55557c5-c68h9" podUID="cc95feca-80e8-41ac-8a29-be0b9c840d1d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:19:46.817348 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:46.817314 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 02:19:49.453324 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:49.453293 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a3a2ac-5d6ff74d96-dnpzn"] Oct 30 02:19:49.485355 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:49.485320 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a3a2ac-6677b6f44c-bpm5m"] Oct 30 02:19:50.145040 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:50.145007 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2d5aa9-698b999554-z8f24"] Oct 30 02:19:50.984324 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:50.984265 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-558327-bd6c69d5b-scg4b" Oct 30 02:19:51.103178 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:51.103149 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/125fc809-a755-460c-8605-363489a85b49-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/125fc809-a755-460c-8605-363489a85b49-plugins-config-volume\") pod \"125fc809-a755-460c-8605-363489a85b49\" (UID: \"125fc809-a755-460c-8605-363489a85b49\") " Oct 30 02:19:51.103389 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:51.103216 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/125fc809-a755-460c-8605-363489a85b49-kube-api-access-h7zr8\" (UniqueName: \"kubernetes.io/projected/125fc809-a755-460c-8605-363489a85b49-kube-api-access-h7zr8\") pod \"125fc809-a755-460c-8605-363489a85b49\" (UID: \"125fc809-a755-460c-8605-363489a85b49\") " Oct 30 02:19:51.103594 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:51.103569 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/125fc809-a755-460c-8605-363489a85b49-plugins-config-volume" pod "125fc809-a755-460c-8605-363489a85b49" (UID: "125fc809-a755-460c-8605-363489a85b49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:19:51.103741 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:51.103721 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/125fc809-a755-460c-8605-363489a85b49-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:51.105487 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:51.105454 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/125fc809-a755-460c-8605-363489a85b49-kube-api-access-h7zr8" pod "125fc809-a755-460c-8605-363489a85b49" (UID: "125fc809-a755-460c-8605-363489a85b49"). InnerVolumeSpecName "kube-api-access-h7zr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:51.110064 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:51.110037 6221 generic.go:356] "Generic (PLEG): container finished" podID="125fc809-a755-460c-8605-363489a85b49" containerID="b8652bab3d6525ebae066d2815fac5993d0c3e10a7347e93eeda9a901338de46" exitCode=0 Oct 30 02:19:51.110164 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:51.110130 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-558327-bd6c69d5b-scg4b" event={"ID":"125fc809-a755-460c-8605-363489a85b49","Type":"ContainerDied","Data":"b8652bab3d6525ebae066d2815fac5993d0c3e10a7347e93eeda9a901338de46"} Oct 30 02:19:51.110486 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:51.110442 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-558327-bd6c69d5b-scg4b" event={"ID":"125fc809-a755-460c-8605-363489a85b49","Type":"ContainerDied","Data":"b8d7fc6136679a97ac007bb424acda23a6d0053c4572d4aa74aa29777c789416"} Oct 30 02:19:51.110708 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:51.110691 6221 scope.go:122] "RemoveContainer" containerID="b8652bab3d6525ebae066d2815fac5993d0c3e10a7347e93eeda9a901338de46" Oct 30 02:19:51.120031 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:51.120007 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-558327-bd6c69d5b-scg4b" Oct 30 02:19:51.129702 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:51.129678 6221 scope.go:122] "RemoveContainer" containerID="b8652bab3d6525ebae066d2815fac5993d0c3e10a7347e93eeda9a901338de46" Oct 30 02:19:51.129995 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:19:51.129975 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8652bab3d6525ebae066d2815fac5993d0c3e10a7347e93eeda9a901338de46\": container with ID starting with b8652bab3d6525ebae066d2815fac5993d0c3e10a7347e93eeda9a901338de46 not found: ID does not exist" containerID="b8652bab3d6525ebae066d2815fac5993d0c3e10a7347e93eeda9a901338de46" Oct 30 02:19:51.130084 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:51.130002 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8652bab3d6525ebae066d2815fac5993d0c3e10a7347e93eeda9a901338de46"} err="failed to get container status \"b8652bab3d6525ebae066d2815fac5993d0c3e10a7347e93eeda9a901338de46\": rpc error: code = NotFound desc = could not find container \"b8652bab3d6525ebae066d2815fac5993d0c3e10a7347e93eeda9a901338de46\": container with ID starting with b8652bab3d6525ebae066d2815fac5993d0c3e10a7347e93eeda9a901338de46 not found: ID does not exist" Oct 30 02:19:51.193366 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:51.193337 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-558327-bd6c69d5b-scg4b"] Oct 30 02:19:51.198921 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:51.198892 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-558327-bd6c69d5b-scg4b"] Oct 30 02:19:51.205091 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:51.205063 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7zr8\" (UniqueName: \"kubernetes.io/projected/125fc809-a755-460c-8605-363489a85b49-kube-api-access-h7zr8\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:51.501240 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:19:51.501200 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 02:19:51.695808 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:51.695746 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d376-76c47844b8-q84n6" Oct 30 02:19:51.749994 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:51.749968 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d376-f998788b7-gvrjq" Oct 30 02:19:51.811945 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:51.811911 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c21d89b-1302-47e9-a329-ff6e4bd5e10e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c21d89b-1302-47e9-a329-ff6e4bd5e10e-plugins-config-volume\") pod \"6c21d89b-1302-47e9-a329-ff6e4bd5e10e\" (UID: \"6c21d89b-1302-47e9-a329-ff6e4bd5e10e\") " Oct 30 02:19:51.812111 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:51.811964 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c21d89b-1302-47e9-a329-ff6e4bd5e10e-kube-api-access-m75cc\" (UniqueName: \"kubernetes.io/projected/6c21d89b-1302-47e9-a329-ff6e4bd5e10e-kube-api-access-m75cc\") pod \"6c21d89b-1302-47e9-a329-ff6e4bd5e10e\" (UID: \"6c21d89b-1302-47e9-a329-ff6e4bd5e10e\") " Oct 30 02:19:51.812111 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:51.811999 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28dc66b5-3755-4151-a2a3-4dbc793d0c28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28dc66b5-3755-4151-a2a3-4dbc793d0c28-plugins-config-volume\") pod \"28dc66b5-3755-4151-a2a3-4dbc793d0c28\" (UID: \"28dc66b5-3755-4151-a2a3-4dbc793d0c28\") " Oct 30 02:19:51.812234 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:51.812139 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28dc66b5-3755-4151-a2a3-4dbc793d0c28-kube-api-access-t78h2\" (UniqueName: \"kubernetes.io/projected/28dc66b5-3755-4151-a2a3-4dbc793d0c28-kube-api-access-t78h2\") pod \"28dc66b5-3755-4151-a2a3-4dbc793d0c28\" (UID: \"28dc66b5-3755-4151-a2a3-4dbc793d0c28\") " Oct 30 02:19:51.812506 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:51.812469 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c21d89b-1302-47e9-a329-ff6e4bd5e10e-plugins-config-volume" pod "6c21d89b-1302-47e9-a329-ff6e4bd5e10e" (UID: "6c21d89b-1302-47e9-a329-ff6e4bd5e10e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:19:51.812506 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:51.812484 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28dc66b5-3755-4151-a2a3-4dbc793d0c28-plugins-config-volume" pod "28dc66b5-3755-4151-a2a3-4dbc793d0c28" (UID: "28dc66b5-3755-4151-a2a3-4dbc793d0c28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:19:51.814331 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:51.814305 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c21d89b-1302-47e9-a329-ff6e4bd5e10e-kube-api-access-m75cc" pod "6c21d89b-1302-47e9-a329-ff6e4bd5e10e" (UID: "6c21d89b-1302-47e9-a329-ff6e4bd5e10e"). InnerVolumeSpecName "kube-api-access-m75cc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:51.814822 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:51.814798 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28dc66b5-3755-4151-a2a3-4dbc793d0c28-kube-api-access-t78h2" pod "28dc66b5-3755-4151-a2a3-4dbc793d0c28" (UID: "28dc66b5-3755-4151-a2a3-4dbc793d0c28"). InnerVolumeSpecName "kube-api-access-t78h2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:51.913455 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:51.913425 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t78h2\" (UniqueName: \"kubernetes.io/projected/28dc66b5-3755-4151-a2a3-4dbc793d0c28-kube-api-access-t78h2\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:51.913455 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:51.913453 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c21d89b-1302-47e9-a329-ff6e4bd5e10e-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:51.913689 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:51.913464 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m75cc\" (UniqueName: \"kubernetes.io/projected/6c21d89b-1302-47e9-a329-ff6e4bd5e10e-kube-api-access-m75cc\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:51.913689 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:51.913474 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28dc66b5-3755-4151-a2a3-4dbc793d0c28-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:52.118715 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:52.118675 6221 generic.go:356] "Generic (PLEG): container finished" podID="6c21d89b-1302-47e9-a329-ff6e4bd5e10e" containerID="a40c22e280ac14b47a74a9c49a6a0a8438e1b46570620095299577cca1f2e3ae" exitCode=0 Oct 30 02:19:52.118715 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:52.118712 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d376-76c47844b8-q84n6" Oct 30 02:19:52.119286 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:52.118754 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d376-76c47844b8-q84n6" event={"ID":"6c21d89b-1302-47e9-a329-ff6e4bd5e10e","Type":"ContainerDied","Data":"a40c22e280ac14b47a74a9c49a6a0a8438e1b46570620095299577cca1f2e3ae"} Oct 30 02:19:52.119286 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:52.118835 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c8d376-76c47844b8-q84n6" event={"ID":"6c21d89b-1302-47e9-a329-ff6e4bd5e10e","Type":"ContainerDied","Data":"f0e7759918a1245cf1f1f15b4f1bcd099e722428a6a0e4ca46f528922b32fb81"} Oct 30 02:19:52.119286 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:52.118858 6221 scope.go:122] "RemoveContainer" containerID="a40c22e280ac14b47a74a9c49a6a0a8438e1b46570620095299577cca1f2e3ae" Oct 30 02:19:52.120360 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:52.120339 6221 generic.go:356] "Generic (PLEG): container finished" podID="28dc66b5-3755-4151-a2a3-4dbc793d0c28" containerID="1e65a93666462732ba6be5470aa969cfdb5fad1a3ec057662f646c46e6680ee6" exitCode=0 Oct 30 02:19:52.120445 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:52.120385 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d376-f998788b7-gvrjq" Oct 30 02:19:52.120445 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:52.120421 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d376-f998788b7-gvrjq" event={"ID":"28dc66b5-3755-4151-a2a3-4dbc793d0c28","Type":"ContainerDied","Data":"1e65a93666462732ba6be5470aa969cfdb5fad1a3ec057662f646c46e6680ee6"} Oct 30 02:19:52.120564 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:52.120464 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d376-f998788b7-gvrjq" event={"ID":"28dc66b5-3755-4151-a2a3-4dbc793d0c28","Type":"ContainerDied","Data":"dfb23d527f9c4766e5c24889c00889421ebace30fd57964afc3ace91471e2eed"} Oct 30 02:19:52.130803 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:52.130624 6221 scope.go:122] "RemoveContainer" containerID="a40c22e280ac14b47a74a9c49a6a0a8438e1b46570620095299577cca1f2e3ae" Oct 30 02:19:52.130935 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:19:52.130915 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a40c22e280ac14b47a74a9c49a6a0a8438e1b46570620095299577cca1f2e3ae\": container with ID starting with a40c22e280ac14b47a74a9c49a6a0a8438e1b46570620095299577cca1f2e3ae not found: ID does not exist" containerID="a40c22e280ac14b47a74a9c49a6a0a8438e1b46570620095299577cca1f2e3ae" Oct 30 02:19:52.131010 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:52.130957 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a40c22e280ac14b47a74a9c49a6a0a8438e1b46570620095299577cca1f2e3ae"} err="failed to get container status \"a40c22e280ac14b47a74a9c49a6a0a8438e1b46570620095299577cca1f2e3ae\": rpc error: code = NotFound desc = could not find container \"a40c22e280ac14b47a74a9c49a6a0a8438e1b46570620095299577cca1f2e3ae\": container with ID starting with a40c22e280ac14b47a74a9c49a6a0a8438e1b46570620095299577cca1f2e3ae not found: ID does not exist" Oct 30 02:19:52.131010 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:52.130982 6221 scope.go:122] "RemoveContainer" containerID="1e65a93666462732ba6be5470aa969cfdb5fad1a3ec057662f646c46e6680ee6" Oct 30 02:19:52.139241 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:52.139216 6221 scope.go:122] "RemoveContainer" containerID="1e65a93666462732ba6be5470aa969cfdb5fad1a3ec057662f646c46e6680ee6" Oct 30 02:19:52.139501 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:19:52.139480 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e65a93666462732ba6be5470aa969cfdb5fad1a3ec057662f646c46e6680ee6\": container with ID starting with 1e65a93666462732ba6be5470aa969cfdb5fad1a3ec057662f646c46e6680ee6 not found: ID does not exist" containerID="1e65a93666462732ba6be5470aa969cfdb5fad1a3ec057662f646c46e6680ee6" Oct 30 02:19:52.139604 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:52.139512 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e65a93666462732ba6be5470aa969cfdb5fad1a3ec057662f646c46e6680ee6"} err="failed to get container status \"1e65a93666462732ba6be5470aa969cfdb5fad1a3ec057662f646c46e6680ee6\": rpc error: code = NotFound desc = could not find container \"1e65a93666462732ba6be5470aa969cfdb5fad1a3ec057662f646c46e6680ee6\": container with ID starting with 1e65a93666462732ba6be5470aa969cfdb5fad1a3ec057662f646c46e6680ee6 not found: ID does not exist" Oct 30 02:19:52.169316 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:52.169289 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d376-f998788b7-gvrjq"] Oct 30 02:19:52.173292 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:52.173267 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c8d376-f998788b7-gvrjq"] Oct 30 02:19:52.206322 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:52.206295 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c8d376-76c47844b8-q84n6"] Oct 30 02:19:52.211886 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:52.211865 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c8d376-76c47844b8-q84n6"] Oct 30 02:19:52.546983 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:52.546951 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="125fc809-a755-460c-8605-363489a85b49" path="/var/lib/kubelet/pods/125fc809-a755-460c-8605-363489a85b49/volumes" Oct 30 02:19:52.547412 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:52.547397 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28dc66b5-3755-4151-a2a3-4dbc793d0c28" path="/var/lib/kubelet/pods/28dc66b5-3755-4151-a2a3-4dbc793d0c28/volumes" Oct 30 02:19:52.547779 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:52.547767 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c21d89b-1302-47e9-a329-ff6e4bd5e10e" path="/var/lib/kubelet/pods/6c21d89b-1302-47e9-a329-ff6e4bd5e10e/volumes" Oct 30 02:19:52.707893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:52.707810 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-558327-7945456798-frsjf"] Oct 30 02:19:52.708349 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:52.708038 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-558327-7945456798-frsjf" podUID="4cbfa236-b599-4625-bf0f-8d5d3e47398e" containerName="maas-api" containerID="cri-o://53d4099edfaf4c0c5620b2809ce001c409af1079d00bef4c1d07d335af9a5668" gracePeriod=30 Oct 30 02:19:52.985443 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:52.985418 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-558327-7945456798-frsjf" Oct 30 02:19:53.130624 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:53.130505 6221 generic.go:356] "Generic (PLEG): container finished" podID="4cbfa236-b599-4625-bf0f-8d5d3e47398e" containerID="53d4099edfaf4c0c5620b2809ce001c409af1079d00bef4c1d07d335af9a5668" exitCode=0 Oct 30 02:19:53.130624 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:53.130591 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-558327-7945456798-frsjf" Oct 30 02:19:53.130624 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:53.130598 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-558327-7945456798-frsjf" event={"ID":"4cbfa236-b599-4625-bf0f-8d5d3e47398e","Type":"ContainerDied","Data":"53d4099edfaf4c0c5620b2809ce001c409af1079d00bef4c1d07d335af9a5668"} Oct 30 02:19:53.131351 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:53.130662 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-558327-7945456798-frsjf" event={"ID":"4cbfa236-b599-4625-bf0f-8d5d3e47398e","Type":"ContainerDied","Data":"6bd7e065505353fd5a24bd32e76279791c80bbf1cfb51f39ce4a7ba177e8835b"} Oct 30 02:19:53.131351 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:53.130695 6221 scope.go:122] "RemoveContainer" containerID="53d4099edfaf4c0c5620b2809ce001c409af1079d00bef4c1d07d335af9a5668" Oct 30 02:19:53.132946 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:53.132573 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cbfa236-b599-4625-bf0f-8d5d3e47398e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cbfa236-b599-4625-bf0f-8d5d3e47398e-maas-api-tls\") pod \"4cbfa236-b599-4625-bf0f-8d5d3e47398e\" (UID: \"4cbfa236-b599-4625-bf0f-8d5d3e47398e\") " Oct 30 02:19:53.132946 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:53.132702 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cbfa236-b599-4625-bf0f-8d5d3e47398e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cbfa236-b599-4625-bf0f-8d5d3e47398e-openshift-service-ca\") pod \"4cbfa236-b599-4625-bf0f-8d5d3e47398e\" (UID: \"4cbfa236-b599-4625-bf0f-8d5d3e47398e\") " Oct 30 02:19:53.132946 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:53.132778 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cbfa236-b599-4625-bf0f-8d5d3e47398e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cbfa236-b599-4625-bf0f-8d5d3e47398e-metrics-certs\") pod \"4cbfa236-b599-4625-bf0f-8d5d3e47398e\" (UID: \"4cbfa236-b599-4625-bf0f-8d5d3e47398e\") " Oct 30 02:19:53.132946 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:53.132886 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cbfa236-b599-4625-bf0f-8d5d3e47398e-kube-api-access-crtj8\" (UniqueName: \"kubernetes.io/projected/4cbfa236-b599-4625-bf0f-8d5d3e47398e-kube-api-access-crtj8\") pod \"4cbfa236-b599-4625-bf0f-8d5d3e47398e\" (UID: \"4cbfa236-b599-4625-bf0f-8d5d3e47398e\") " Oct 30 02:19:53.133183 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:53.133159 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cbfa236-b599-4625-bf0f-8d5d3e47398e-openshift-service-ca" pod "4cbfa236-b599-4625-bf0f-8d5d3e47398e" (UID: "4cbfa236-b599-4625-bf0f-8d5d3e47398e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:19:53.133393 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:53.133372 6221 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cbfa236-b599-4625-bf0f-8d5d3e47398e-openshift-service-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:53.134872 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:53.134847 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cbfa236-b599-4625-bf0f-8d5d3e47398e-maas-api-tls" pod "4cbfa236-b599-4625-bf0f-8d5d3e47398e" (UID: "4cbfa236-b599-4625-bf0f-8d5d3e47398e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:19:53.134977 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:53.134918 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cbfa236-b599-4625-bf0f-8d5d3e47398e-kube-api-access-crtj8" pod "4cbfa236-b599-4625-bf0f-8d5d3e47398e" (UID: "4cbfa236-b599-4625-bf0f-8d5d3e47398e"). InnerVolumeSpecName "kube-api-access-crtj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:53.135049 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:53.135012 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cbfa236-b599-4625-bf0f-8d5d3e47398e-metrics-certs" pod "4cbfa236-b599-4625-bf0f-8d5d3e47398e" (UID: "4cbfa236-b599-4625-bf0f-8d5d3e47398e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:19:53.159103 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:53.159069 6221 scope.go:122] "RemoveContainer" containerID="53d4099edfaf4c0c5620b2809ce001c409af1079d00bef4c1d07d335af9a5668" Oct 30 02:19:53.159568 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:19:53.159476 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53d4099edfaf4c0c5620b2809ce001c409af1079d00bef4c1d07d335af9a5668\": container with ID starting with 53d4099edfaf4c0c5620b2809ce001c409af1079d00bef4c1d07d335af9a5668 not found: ID does not exist" containerID="53d4099edfaf4c0c5620b2809ce001c409af1079d00bef4c1d07d335af9a5668" Oct 30 02:19:53.159568 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:53.159518 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53d4099edfaf4c0c5620b2809ce001c409af1079d00bef4c1d07d335af9a5668"} err="failed to get container status \"53d4099edfaf4c0c5620b2809ce001c409af1079d00bef4c1d07d335af9a5668\": rpc error: code = NotFound desc = could not find container \"53d4099edfaf4c0c5620b2809ce001c409af1079d00bef4c1d07d335af9a5668\": container with ID starting with 53d4099edfaf4c0c5620b2809ce001c409af1079d00bef4c1d07d335af9a5668 not found: ID does not exist" Oct 30 02:19:53.234613 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:53.234576 6221 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4cbfa236-b599-4625-bf0f-8d5d3e47398e-maas-api-tls\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:53.234788 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:53.234629 6221 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4cbfa236-b599-4625-bf0f-8d5d3e47398e-metrics-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:53.234788 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:53.234639 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crtj8\" (UniqueName: \"kubernetes.io/projected/4cbfa236-b599-4625-bf0f-8d5d3e47398e-kube-api-access-crtj8\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:53.505517 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:53.505472 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-558327-7945456798-frsjf"] Oct 30 02:19:53.534178 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:53.534144 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-558327-7945456798-frsjf"] Oct 30 02:19:54.453662 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:54.453621 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-a3a2ac-5d6ff74d96-dnpzn" podUID="702e03b2-6e41-4a27-837a-ede7f1c2e2c2" containerName="payload-processing" containerID="cri-o://dc649fe4c29d4979c29690108ae04f89428ace55d60d815535b1f72a8e0df35f" gracePeriod=25 Oct 30 02:19:54.486607 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:54.486566 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a3a2ac-6677b6f44c-bpm5m" podUID="26457d5d-6b8f-4e85-8bc1-321115791efc" containerName="payload-pre-processing" containerID="cri-o://cf1461e454b5a0ca17822203ff258a06f43a988b751fd9de7cfae531a4ff22af" gracePeriod=25 Oct 30 02:19:54.547200 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:54.547170 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cbfa236-b599-4625-bf0f-8d5d3e47398e" path="/var/lib/kubelet/pods/4cbfa236-b599-4625-bf0f-8d5d3e47398e/volumes" Oct 30 02:19:55.149632 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:55.145594 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-2d5aa9-698b999554-z8f24" podUID="27948676-86ab-49f1-a4f7-3083a591738c" containerName="payload-processing" containerID="cri-o://c67ae1db4c5cb414f76612649870aa110c0597a074adf8d0251e7b75cb1acd98" gracePeriod=25 Oct 30 02:19:55.320740 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:55.320678 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-340ae6-8f55557c5-c68h9" Oct 30 02:19:55.417110 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:55.417085 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2d5aa9-698b999554-z8f24" Oct 30 02:19:55.461629 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:55.461600 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc95feca-80e8-41ac-8a29-be0b9c840d1d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc95feca-80e8-41ac-8a29-be0b9c840d1d-plugins-config-volume\") pod \"cc95feca-80e8-41ac-8a29-be0b9c840d1d\" (UID: \"cc95feca-80e8-41ac-8a29-be0b9c840d1d\") " Oct 30 02:19:55.462146 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:55.461751 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc95feca-80e8-41ac-8a29-be0b9c840d1d-kube-api-access-mjg7r\" (UniqueName: \"kubernetes.io/projected/cc95feca-80e8-41ac-8a29-be0b9c840d1d-kube-api-access-mjg7r\") pod \"cc95feca-80e8-41ac-8a29-be0b9c840d1d\" (UID: \"cc95feca-80e8-41ac-8a29-be0b9c840d1d\") " Oct 30 02:19:55.462146 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:55.462006 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc95feca-80e8-41ac-8a29-be0b9c840d1d-plugins-config-volume" pod "cc95feca-80e8-41ac-8a29-be0b9c840d1d" (UID: "cc95feca-80e8-41ac-8a29-be0b9c840d1d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:19:55.464027 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:55.464001 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc95feca-80e8-41ac-8a29-be0b9c840d1d-kube-api-access-mjg7r" pod "cc95feca-80e8-41ac-8a29-be0b9c840d1d" (UID: "cc95feca-80e8-41ac-8a29-be0b9c840d1d"). InnerVolumeSpecName "kube-api-access-mjg7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:55.562837 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:55.562801 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27948676-86ab-49f1-a4f7-3083a591738c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27948676-86ab-49f1-a4f7-3083a591738c-plugins-config-volume\") pod \"27948676-86ab-49f1-a4f7-3083a591738c\" (UID: \"27948676-86ab-49f1-a4f7-3083a591738c\") " Oct 30 02:19:55.563069 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:55.563050 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27948676-86ab-49f1-a4f7-3083a591738c-kube-api-access-mlwdv\" (UniqueName: \"kubernetes.io/projected/27948676-86ab-49f1-a4f7-3083a591738c-kube-api-access-mlwdv\") pod \"27948676-86ab-49f1-a4f7-3083a591738c\" (UID: \"27948676-86ab-49f1-a4f7-3083a591738c\") " Oct 30 02:19:55.563258 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:55.563238 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27948676-86ab-49f1-a4f7-3083a591738c-plugins-config-volume" pod "27948676-86ab-49f1-a4f7-3083a591738c" (UID: "27948676-86ab-49f1-a4f7-3083a591738c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:19:55.563403 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:55.563388 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjg7r\" (UniqueName: \"kubernetes.io/projected/cc95feca-80e8-41ac-8a29-be0b9c840d1d-kube-api-access-mjg7r\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:55.563439 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:55.563407 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27948676-86ab-49f1-a4f7-3083a591738c-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:55.563439 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:55.563417 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc95feca-80e8-41ac-8a29-be0b9c840d1d-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:55.565073 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:55.565052 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27948676-86ab-49f1-a4f7-3083a591738c-kube-api-access-mlwdv" pod "27948676-86ab-49f1-a4f7-3083a591738c" (UID: "27948676-86ab-49f1-a4f7-3083a591738c"). InnerVolumeSpecName "kube-api-access-mlwdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:55.664601 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:55.664502 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlwdv\" (UniqueName: \"kubernetes.io/projected/27948676-86ab-49f1-a4f7-3083a591738c-kube-api-access-mlwdv\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:56.161228 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:56.161192 6221 generic.go:356] "Generic (PLEG): container finished" podID="cc95feca-80e8-41ac-8a29-be0b9c840d1d" containerID="6062f35df1eabd36ad3cf281db91d864e1c1cc759bf12b732fde5a74cc85ac07" exitCode=0 Oct 30 02:19:56.161429 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:56.161242 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-340ae6-8f55557c5-c68h9" Oct 30 02:19:56.161429 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:56.161284 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-340ae6-8f55557c5-c68h9" event={"ID":"cc95feca-80e8-41ac-8a29-be0b9c840d1d","Type":"ContainerDied","Data":"6062f35df1eabd36ad3cf281db91d864e1c1cc759bf12b732fde5a74cc85ac07"} Oct 30 02:19:56.161429 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:56.161354 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-340ae6-8f55557c5-c68h9" event={"ID":"cc95feca-80e8-41ac-8a29-be0b9c840d1d","Type":"ContainerDied","Data":"15ca870a50a1926fe1529ecdd5a72018dc873d2cd40c81b4f090d165c4e9753d"} Oct 30 02:19:56.161429 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:56.161385 6221 scope.go:122] "RemoveContainer" containerID="6062f35df1eabd36ad3cf281db91d864e1c1cc759bf12b732fde5a74cc85ac07" Oct 30 02:19:56.162833 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:56.162640 6221 generic.go:356] "Generic (PLEG): container finished" podID="27948676-86ab-49f1-a4f7-3083a591738c" containerID="c67ae1db4c5cb414f76612649870aa110c0597a074adf8d0251e7b75cb1acd98" exitCode=0 Oct 30 02:19:56.162833 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:56.162686 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2d5aa9-698b999554-z8f24" Oct 30 02:19:56.162833 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:56.162722 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2d5aa9-698b999554-z8f24" event={"ID":"27948676-86ab-49f1-a4f7-3083a591738c","Type":"ContainerDied","Data":"c67ae1db4c5cb414f76612649870aa110c0597a074adf8d0251e7b75cb1acd98"} Oct 30 02:19:56.162833 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:56.162776 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2d5aa9-698b999554-z8f24" event={"ID":"27948676-86ab-49f1-a4f7-3083a591738c","Type":"ContainerDied","Data":"201ee2c2b5e779e7ca863aa65aa7924ee30ffa42e1d214f954775128ae4a5017"} Oct 30 02:19:56.171502 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:56.171483 6221 scope.go:122] "RemoveContainer" containerID="6062f35df1eabd36ad3cf281db91d864e1c1cc759bf12b732fde5a74cc85ac07" Oct 30 02:19:56.171802 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:19:56.171780 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6062f35df1eabd36ad3cf281db91d864e1c1cc759bf12b732fde5a74cc85ac07\": container with ID starting with 6062f35df1eabd36ad3cf281db91d864e1c1cc759bf12b732fde5a74cc85ac07 not found: ID does not exist" containerID="6062f35df1eabd36ad3cf281db91d864e1c1cc759bf12b732fde5a74cc85ac07" Oct 30 02:19:56.171887 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:56.171808 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6062f35df1eabd36ad3cf281db91d864e1c1cc759bf12b732fde5a74cc85ac07"} err="failed to get container status \"6062f35df1eabd36ad3cf281db91d864e1c1cc759bf12b732fde5a74cc85ac07\": rpc error: code = NotFound desc = could not find container \"6062f35df1eabd36ad3cf281db91d864e1c1cc759bf12b732fde5a74cc85ac07\": container with ID starting with 6062f35df1eabd36ad3cf281db91d864e1c1cc759bf12b732fde5a74cc85ac07 not found: ID does not exist" Oct 30 02:19:56.171887 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:56.171826 6221 scope.go:122] "RemoveContainer" containerID="c67ae1db4c5cb414f76612649870aa110c0597a074adf8d0251e7b75cb1acd98" Oct 30 02:19:56.180889 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:56.180873 6221 scope.go:122] "RemoveContainer" containerID="c67ae1db4c5cb414f76612649870aa110c0597a074adf8d0251e7b75cb1acd98" Oct 30 02:19:56.181142 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:19:56.181124 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c67ae1db4c5cb414f76612649870aa110c0597a074adf8d0251e7b75cb1acd98\": container with ID starting with c67ae1db4c5cb414f76612649870aa110c0597a074adf8d0251e7b75cb1acd98 not found: ID does not exist" containerID="c67ae1db4c5cb414f76612649870aa110c0597a074adf8d0251e7b75cb1acd98" Oct 30 02:19:56.181206 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:56.181153 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c67ae1db4c5cb414f76612649870aa110c0597a074adf8d0251e7b75cb1acd98"} err="failed to get container status \"c67ae1db4c5cb414f76612649870aa110c0597a074adf8d0251e7b75cb1acd98\": rpc error: code = NotFound desc = could not find container \"c67ae1db4c5cb414f76612649870aa110c0597a074adf8d0251e7b75cb1acd98\": container with ID starting with c67ae1db4c5cb414f76612649870aa110c0597a074adf8d0251e7b75cb1acd98 not found: ID does not exist" Oct 30 02:19:56.212671 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:56.212638 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2d5aa9-698b999554-z8f24"] Oct 30 02:19:56.225735 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:56.225707 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2d5aa9-698b999554-z8f24"] Oct 30 02:19:56.254526 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:56.254496 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-340ae6-8f55557c5-c68h9"] Oct 30 02:19:56.259050 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:56.259021 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-340ae6-8f55557c5-c68h9"] Oct 30 02:19:56.550488 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:56.550454 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27948676-86ab-49f1-a4f7-3083a591738c" path="/var/lib/kubelet/pods/27948676-86ab-49f1-a4f7-3083a591738c/volumes" Oct 30 02:19:56.551075 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:56.551052 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc95feca-80e8-41ac-8a29-be0b9c840d1d" path="/var/lib/kubelet/pods/cc95feca-80e8-41ac-8a29-be0b9c840d1d/volumes" Oct 30 02:19:58.307909 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:58.307875 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2d5aa9-856559788f-hv2fv"] Oct 30 02:19:58.308416 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:58.308108 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2d5aa9-856559788f-hv2fv" podUID="fbab4591-a274-4c63-9004-8c05dd8676df" containerName="maas-api" containerID="cri-o://54633b5ade30d3be7aa012c92e47606beaccb118a05577ce67bd715d4dfa4cd9" gracePeriod=30 Oct 30 02:19:58.613308 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:58.613281 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2d5aa9-856559788f-hv2fv" Oct 30 02:19:58.627243 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:58.627213 6221 scope.go:122] "RemoveContainer" containerID="45cafb0ca65d1e7fc8c0a6163644325373636108311345d2b64ccd12ce8ef91a" Oct 30 02:19:58.699457 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:58.699420 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbab4591-a274-4c63-9004-8c05dd8676df-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbab4591-a274-4c63-9004-8c05dd8676df-maas-api-tls\") pod \"fbab4591-a274-4c63-9004-8c05dd8676df\" (UID: \"fbab4591-a274-4c63-9004-8c05dd8676df\") " Oct 30 02:19:58.699684 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:58.699521 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbab4591-a274-4c63-9004-8c05dd8676df-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbab4591-a274-4c63-9004-8c05dd8676df-openshift-service-ca\") pod \"fbab4591-a274-4c63-9004-8c05dd8676df\" (UID: \"fbab4591-a274-4c63-9004-8c05dd8676df\") " Oct 30 02:19:58.699774 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:58.699691 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbab4591-a274-4c63-9004-8c05dd8676df-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbab4591-a274-4c63-9004-8c05dd8676df-metrics-certs\") pod \"fbab4591-a274-4c63-9004-8c05dd8676df\" (UID: \"fbab4591-a274-4c63-9004-8c05dd8676df\") " Oct 30 02:19:58.699856 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:58.699791 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbab4591-a274-4c63-9004-8c05dd8676df-kube-api-access-9d7tl\" (UniqueName: \"kubernetes.io/projected/fbab4591-a274-4c63-9004-8c05dd8676df-kube-api-access-9d7tl\") pod \"fbab4591-a274-4c63-9004-8c05dd8676df\" (UID: \"fbab4591-a274-4c63-9004-8c05dd8676df\") " Oct 30 02:19:58.701003 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:58.700973 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbab4591-a274-4c63-9004-8c05dd8676df-openshift-service-ca" pod "fbab4591-a274-4c63-9004-8c05dd8676df" (UID: "fbab4591-a274-4c63-9004-8c05dd8676df"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:19:58.703285 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:58.703254 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbab4591-a274-4c63-9004-8c05dd8676df-kube-api-access-9d7tl" pod "fbab4591-a274-4c63-9004-8c05dd8676df" (UID: "fbab4591-a274-4c63-9004-8c05dd8676df"). InnerVolumeSpecName "kube-api-access-9d7tl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:58.706573 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:58.706162 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbab4591-a274-4c63-9004-8c05dd8676df-metrics-certs" pod "fbab4591-a274-4c63-9004-8c05dd8676df" (UID: "fbab4591-a274-4c63-9004-8c05dd8676df"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:19:58.706573 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:58.706205 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbab4591-a274-4c63-9004-8c05dd8676df-maas-api-tls" pod "fbab4591-a274-4c63-9004-8c05dd8676df" (UID: "fbab4591-a274-4c63-9004-8c05dd8676df"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:19:58.797159 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:58.797120 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" podUID="018c3fe2-3e67-45ed-a3d7-d70f3fe5b956" containerName="main" containerID="cri-o://48289d04014d15ff43e0a04e923abf65a2f96480856367ffb54716448d8ac737" gracePeriod=45 Oct 30 02:19:58.801128 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:58.801093 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d7tl\" (UniqueName: \"kubernetes.io/projected/fbab4591-a274-4c63-9004-8c05dd8676df-kube-api-access-9d7tl\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:58.801128 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:58.801126 6221 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbab4591-a274-4c63-9004-8c05dd8676df-maas-api-tls\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:58.801285 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:58.801136 6221 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbab4591-a274-4c63-9004-8c05dd8676df-openshift-service-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:58.801285 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:58.801145 6221 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbab4591-a274-4c63-9004-8c05dd8676df-metrics-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:59.086398 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:59.086357 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" Oct 30 02:19:59.185367 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:59.185285 6221 generic.go:356] "Generic (PLEG): container finished" podID="fbab4591-a274-4c63-9004-8c05dd8676df" containerID="54633b5ade30d3be7aa012c92e47606beaccb118a05577ce67bd715d4dfa4cd9" exitCode=0 Oct 30 02:19:59.185367 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:59.185356 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2d5aa9-856559788f-hv2fv" Oct 30 02:19:59.185657 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:59.185366 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2d5aa9-856559788f-hv2fv" event={"ID":"fbab4591-a274-4c63-9004-8c05dd8676df","Type":"ContainerDied","Data":"54633b5ade30d3be7aa012c92e47606beaccb118a05577ce67bd715d4dfa4cd9"} Oct 30 02:19:59.185657 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:59.185422 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-2d5aa9-856559788f-hv2fv" event={"ID":"fbab4591-a274-4c63-9004-8c05dd8676df","Type":"ContainerDied","Data":"4aad5ffac3e294a7a136a10c33c71bb72c4efd9ea79fea3d32d095499863f539"} Oct 30 02:19:59.185657 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:59.185463 6221 scope.go:122] "RemoveContainer" containerID="54633b5ade30d3be7aa012c92e47606beaccb118a05577ce67bd715d4dfa4cd9" Oct 30 02:19:59.187319 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:59.187288 6221 generic.go:356] "Generic (PLEG): container finished" podID="018c3fe2-3e67-45ed-a3d7-d70f3fe5b956" containerID="48289d04014d15ff43e0a04e923abf65a2f96480856367ffb54716448d8ac737" exitCode=0 Oct 30 02:19:59.187456 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:59.187332 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" Oct 30 02:19:59.187456 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:59.187423 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" event={"ID":"018c3fe2-3e67-45ed-a3d7-d70f3fe5b956","Type":"ContainerDied","Data":"48289d04014d15ff43e0a04e923abf65a2f96480856367ffb54716448d8ac737"} Oct 30 02:19:59.187615 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:59.187457 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl" event={"ID":"018c3fe2-3e67-45ed-a3d7-d70f3fe5b956","Type":"ContainerDied","Data":"2f29ede5aef7e955a1072fcff850ee1b354808104ebeb00e53ec3002b69fc2d2"} Oct 30 02:19:59.198388 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:59.198366 6221 scope.go:122] "RemoveContainer" containerID="54633b5ade30d3be7aa012c92e47606beaccb118a05577ce67bd715d4dfa4cd9" Oct 30 02:19:59.198751 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:19:59.198722 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54633b5ade30d3be7aa012c92e47606beaccb118a05577ce67bd715d4dfa4cd9\": container with ID starting with 54633b5ade30d3be7aa012c92e47606beaccb118a05577ce67bd715d4dfa4cd9 not found: ID does not exist" containerID="54633b5ade30d3be7aa012c92e47606beaccb118a05577ce67bd715d4dfa4cd9" Oct 30 02:19:59.198818 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:59.198762 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54633b5ade30d3be7aa012c92e47606beaccb118a05577ce67bd715d4dfa4cd9"} err="failed to get container status \"54633b5ade30d3be7aa012c92e47606beaccb118a05577ce67bd715d4dfa4cd9\": rpc error: code = NotFound desc = could not find container \"54633b5ade30d3be7aa012c92e47606beaccb118a05577ce67bd715d4dfa4cd9\": container with ID starting with 54633b5ade30d3be7aa012c92e47606beaccb118a05577ce67bd715d4dfa4cd9 not found: ID does not exist" Oct 30 02:19:59.198818 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:59.198787 6221 scope.go:122] "RemoveContainer" containerID="48289d04014d15ff43e0a04e923abf65a2f96480856367ffb54716448d8ac737" Oct 30 02:19:59.206418 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:59.206396 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/018c3fe2-3e67-45ed-a3d7-d70f3fe5b956-dshm\" (UniqueName: \"kubernetes.io/empty-dir/018c3fe2-3e67-45ed-a3d7-d70f3fe5b956-dshm\") pod \"018c3fe2-3e67-45ed-a3d7-d70f3fe5b956\" (UID: \"018c3fe2-3e67-45ed-a3d7-d70f3fe5b956\") " Oct 30 02:19:59.206640 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:59.206619 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/018c3fe2-3e67-45ed-a3d7-d70f3fe5b956-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/018c3fe2-3e67-45ed-a3d7-d70f3fe5b956-tmp-dir\") pod \"018c3fe2-3e67-45ed-a3d7-d70f3fe5b956\" (UID: \"018c3fe2-3e67-45ed-a3d7-d70f3fe5b956\") " Oct 30 02:19:59.206795 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:59.206679 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/018c3fe2-3e67-45ed-a3d7-d70f3fe5b956-kube-api-access-sffj4\" (UniqueName: \"kubernetes.io/projected/018c3fe2-3e67-45ed-a3d7-d70f3fe5b956-kube-api-access-sffj4\") pod \"018c3fe2-3e67-45ed-a3d7-d70f3fe5b956\" (UID: \"018c3fe2-3e67-45ed-a3d7-d70f3fe5b956\") " Oct 30 02:19:59.206795 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:59.206718 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/018c3fe2-3e67-45ed-a3d7-d70f3fe5b956-tls-certs\" (UniqueName: \"kubernetes.io/secret/018c3fe2-3e67-45ed-a3d7-d70f3fe5b956-tls-certs\") pod \"018c3fe2-3e67-45ed-a3d7-d70f3fe5b956\" (UID: \"018c3fe2-3e67-45ed-a3d7-d70f3fe5b956\") " Oct 30 02:19:59.206795 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:59.206777 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/018c3fe2-3e67-45ed-a3d7-d70f3fe5b956-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/018c3fe2-3e67-45ed-a3d7-d70f3fe5b956-model-cache\") pod \"018c3fe2-3e67-45ed-a3d7-d70f3fe5b956\" (UID: \"018c3fe2-3e67-45ed-a3d7-d70f3fe5b956\") " Oct 30 02:19:59.206961 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:59.206839 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/018c3fe2-3e67-45ed-a3d7-d70f3fe5b956-home\" (UniqueName: \"kubernetes.io/empty-dir/018c3fe2-3e67-45ed-a3d7-d70f3fe5b956-home\") pod \"018c3fe2-3e67-45ed-a3d7-d70f3fe5b956\" (UID: \"018c3fe2-3e67-45ed-a3d7-d70f3fe5b956\") " Oct 30 02:19:59.207017 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:59.206974 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/018c3fe2-3e67-45ed-a3d7-d70f3fe5b956-tmp-dir" pod "018c3fe2-3e67-45ed-a3d7-d70f3fe5b956" (UID: "018c3fe2-3e67-45ed-a3d7-d70f3fe5b956"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:59.207367 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:59.207342 6221 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/018c3fe2-3e67-45ed-a3d7-d70f3fe5b956-tmp-dir\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:59.207598 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:59.207576 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/018c3fe2-3e67-45ed-a3d7-d70f3fe5b956-home" pod "018c3fe2-3e67-45ed-a3d7-d70f3fe5b956" (UID: "018c3fe2-3e67-45ed-a3d7-d70f3fe5b956"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:59.207798 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:59.207777 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/018c3fe2-3e67-45ed-a3d7-d70f3fe5b956-model-cache" pod "018c3fe2-3e67-45ed-a3d7-d70f3fe5b956" (UID: "018c3fe2-3e67-45ed-a3d7-d70f3fe5b956"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:59.208915 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:59.208890 6221 scope.go:122] "RemoveContainer" containerID="48289d04014d15ff43e0a04e923abf65a2f96480856367ffb54716448d8ac737" Oct 30 02:19:59.209232 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:19:59.209209 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48289d04014d15ff43e0a04e923abf65a2f96480856367ffb54716448d8ac737\": container with ID starting with 48289d04014d15ff43e0a04e923abf65a2f96480856367ffb54716448d8ac737 not found: ID does not exist" containerID="48289d04014d15ff43e0a04e923abf65a2f96480856367ffb54716448d8ac737" Oct 30 02:19:59.209329 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:59.209238 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/018c3fe2-3e67-45ed-a3d7-d70f3fe5b956-kube-api-access-sffj4" pod "018c3fe2-3e67-45ed-a3d7-d70f3fe5b956" (UID: "018c3fe2-3e67-45ed-a3d7-d70f3fe5b956"). InnerVolumeSpecName "kube-api-access-sffj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:19:59.209329 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:59.209250 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48289d04014d15ff43e0a04e923abf65a2f96480856367ffb54716448d8ac737"} err="failed to get container status \"48289d04014d15ff43e0a04e923abf65a2f96480856367ffb54716448d8ac737\": rpc error: code = NotFound desc = could not find container \"48289d04014d15ff43e0a04e923abf65a2f96480856367ffb54716448d8ac737\": container with ID starting with 48289d04014d15ff43e0a04e923abf65a2f96480856367ffb54716448d8ac737 not found: ID does not exist" Oct 30 02:19:59.209517 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:59.209497 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/018c3fe2-3e67-45ed-a3d7-d70f3fe5b956-dshm" pod "018c3fe2-3e67-45ed-a3d7-d70f3fe5b956" (UID: "018c3fe2-3e67-45ed-a3d7-d70f3fe5b956"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:19:59.210180 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:59.210160 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/018c3fe2-3e67-45ed-a3d7-d70f3fe5b956-tls-certs" pod "018c3fe2-3e67-45ed-a3d7-d70f3fe5b956" (UID: "018c3fe2-3e67-45ed-a3d7-d70f3fe5b956"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:19:59.246750 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:59.246710 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2d5aa9-856559788f-hv2fv"] Oct 30 02:19:59.246959 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:59.246771 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-2d5aa9-856559788f-hv2fv"] Oct 30 02:19:59.308281 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:59.308248 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sffj4\" (UniqueName: \"kubernetes.io/projected/018c3fe2-3e67-45ed-a3d7-d70f3fe5b956-kube-api-access-sffj4\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:59.308281 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:59.308277 6221 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/018c3fe2-3e67-45ed-a3d7-d70f3fe5b956-tls-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:59.308730 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:59.308291 6221 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/018c3fe2-3e67-45ed-a3d7-d70f3fe5b956-model-cache\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:59.308730 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:59.308299 6221 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/018c3fe2-3e67-45ed-a3d7-d70f3fe5b956-home\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:59.308730 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:59.308308 6221 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/018c3fe2-3e67-45ed-a3d7-d70f3fe5b956-dshm\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:19:59.540037 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:59.540004 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl"] Oct 30 02:19:59.552707 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:19:59.552681 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-a3a2ac/premium-w0-a3a2ac-kserve-5bd775bbf9-sshgl"] Oct 30 02:20:00.546395 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:00.546357 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="018c3fe2-3e67-45ed-a3d7-d70f3fe5b956" path="/var/lib/kubelet/pods/018c3fe2-3e67-45ed-a3d7-d70f3fe5b956/volumes" Oct 30 02:20:00.546837 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:00.546817 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbab4591-a274-4c63-9004-8c05dd8676df" path="/var/lib/kubelet/pods/fbab4591-a274-4c63-9004-8c05dd8676df/volumes" Oct 30 02:20:00.928006 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:00.927916 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz"] Oct 30 02:20:00.928200 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:00.928175 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" podUID="666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa" containerName="istio-proxy" containerID="cri-o://dc349ecb606a610d09839aee538c745dadcc54be91965eb1bd2c6531ca741995" gracePeriod=30 Oct 30 02:20:01.150608 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:01.150564 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-a3a2ac-5d6ff74d96-dnpzn" podUID="702e03b2-6e41-4a27-837a-ede7f1c2e2c2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:20:01.150810 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:01.150571 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a3a2ac-6677b6f44c-bpm5m" podUID="26457d5d-6b8f-4e85-8bc1-321115791efc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:20:05.571541 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:05.571490 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 02:20:06.205346 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:06.205275 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" Oct 30 02:20:06.227357 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:06.227323 6221 generic.go:356] "Generic (PLEG): container finished" podID="666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa" containerID="dc349ecb606a610d09839aee538c745dadcc54be91965eb1bd2c6531ca741995" exitCode=0 Oct 30 02:20:06.227556 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:06.227373 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" Oct 30 02:20:06.227556 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:06.227383 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" event={"ID":"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa","Type":"ContainerDied","Data":"dc349ecb606a610d09839aee538c745dadcc54be91965eb1bd2c6531ca741995"} Oct 30 02:20:06.227556 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:06.227431 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz" event={"ID":"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa","Type":"ContainerDied","Data":"c175d9c16a4645d44873c2543f9e0758d1746a068181d7a7150f2060d9aea532"} Oct 30 02:20:06.227556 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:06.227453 6221 scope.go:122] "RemoveContainer" containerID="dc349ecb606a610d09839aee538c745dadcc54be91965eb1bd2c6531ca741995" Oct 30 02:20:06.236642 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:06.236621 6221 scope.go:122] "RemoveContainer" containerID="fc38a75f406fb4f803c813cf9685c74f1f5b1e2808a01db945239b2d6de1db4f" Oct 30 02:20:06.245113 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:06.245091 6221 scope.go:122] "RemoveContainer" containerID="dc349ecb606a610d09839aee538c745dadcc54be91965eb1bd2c6531ca741995" Oct 30 02:20:06.245393 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:20:06.245374 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc349ecb606a610d09839aee538c745dadcc54be91965eb1bd2c6531ca741995\": container with ID starting with dc349ecb606a610d09839aee538c745dadcc54be91965eb1bd2c6531ca741995 not found: ID does not exist" containerID="dc349ecb606a610d09839aee538c745dadcc54be91965eb1bd2c6531ca741995" Oct 30 02:20:06.245448 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:06.245403 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc349ecb606a610d09839aee538c745dadcc54be91965eb1bd2c6531ca741995"} err="failed to get container status \"dc349ecb606a610d09839aee538c745dadcc54be91965eb1bd2c6531ca741995\": rpc error: code = NotFound desc = could not find container \"dc349ecb606a610d09839aee538c745dadcc54be91965eb1bd2c6531ca741995\": container with ID starting with dc349ecb606a610d09839aee538c745dadcc54be91965eb1bd2c6531ca741995 not found: ID does not exist" Oct 30 02:20:06.245448 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:06.245421 6221 scope.go:122] "RemoveContainer" containerID="fc38a75f406fb4f803c813cf9685c74f1f5b1e2808a01db945239b2d6de1db4f" Oct 30 02:20:06.245786 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:20:06.245766 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc38a75f406fb4f803c813cf9685c74f1f5b1e2808a01db945239b2d6de1db4f\": container with ID starting with fc38a75f406fb4f803c813cf9685c74f1f5b1e2808a01db945239b2d6de1db4f not found: ID does not exist" containerID="fc38a75f406fb4f803c813cf9685c74f1f5b1e2808a01db945239b2d6de1db4f" Oct 30 02:20:06.245853 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:06.245795 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc38a75f406fb4f803c813cf9685c74f1f5b1e2808a01db945239b2d6de1db4f"} err="failed to get container status \"fc38a75f406fb4f803c813cf9685c74f1f5b1e2808a01db945239b2d6de1db4f\": rpc error: code = NotFound desc = could not find container \"fc38a75f406fb4f803c813cf9685c74f1f5b1e2808a01db945239b2d6de1db4f\": container with ID starting with fc38a75f406fb4f803c813cf9685c74f1f5b1e2808a01db945239b2d6de1db4f not found: ID does not exist" Oct 30 02:20:06.387269 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:06.387229 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-workload-socket\") pod \"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa\" (UID: \"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa\") " Oct 30 02:20:06.387485 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:06.387287 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-kube-api-access-qgrx9\" (UniqueName: \"kubernetes.io/projected/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-kube-api-access-qgrx9\") pod \"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa\" (UID: \"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa\") " Oct 30 02:20:06.387485 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:06.387322 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-istio-data\") pod \"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa\" (UID: \"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa\") " Oct 30 02:20:06.387485 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:06.387345 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-istiod-ca-cert\") pod \"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa\" (UID: \"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa\") " Oct 30 02:20:06.387485 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:06.387430 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-credential-socket\") pod \"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa\" (UID: \"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa\") " Oct 30 02:20:06.387485 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:06.387456 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-workload-certs\") pod \"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa\" (UID: \"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa\") " Oct 30 02:20:06.387798 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:06.387508 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-istio-envoy\") pod \"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa\" (UID: \"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa\") " Oct 30 02:20:06.387798 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:06.387559 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-workload-socket" pod "666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa" (UID: "666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:20:06.387798 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:06.387568 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-istio-podinfo\") pod \"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa\" (UID: \"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa\") " Oct 30 02:20:06.387798 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:06.387638 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-istio-data" pod "666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa" (UID: "666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:20:06.387798 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:06.387698 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-istio-token\" (UniqueName: \"kubernetes.io/projected/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-istio-token\") pod \"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa\" (UID: \"666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa\") " Oct 30 02:20:06.388057 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:06.387865 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-credential-socket" pod "666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa" (UID: "666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:20:06.388057 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:06.387883 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-workload-certs" pod "666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa" (UID: "666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:20:06.388057 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:06.387956 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-istiod-ca-cert" pod "666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa" (UID: "666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:20:06.388237 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:06.388153 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:20:06.388237 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:06.388171 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:20:06.388237 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:06.388184 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:20:06.388237 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:06.388197 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:20:06.388237 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:06.388205 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:20:06.389962 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:06.389937 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-kube-api-access-qgrx9" pod "666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa" (UID: "666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa"). InnerVolumeSpecName "kube-api-access-qgrx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:20:06.390100 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:06.390038 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-istio-token" pod "666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa" (UID: "666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:20:06.390201 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:06.390180 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-istio-envoy" pod "666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa" (UID: "666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:20:06.390201 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:06.390184 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-istio-podinfo" pod "666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa" (UID: "666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:20:06.489719 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:06.489633 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgrx9\" (UniqueName: \"kubernetes.io/projected/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-kube-api-access-qgrx9\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:20:06.489719 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:06.489667 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:20:06.489719 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:06.489676 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:20:06.489719 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:06.489684 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:20:06.596996 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:06.596962 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz"] Oct 30 02:20:06.604854 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:06.604822 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-558327-openshift-default-576847689f-zwshz"] Oct 30 02:20:07.004486 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:07.004450 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a3a2ac-8456757f5d-8xzjx"] Oct 30 02:20:07.004763 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:07.004723 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a3a2ac-8456757f5d-8xzjx" podUID="24f7ec30-91bf-4af8-a913-f1650e340227" containerName="maas-api" containerID="cri-o://848d0f4e8d712e821cbfb0152c347aa883bff13b8400939a4da334a8219e2c62" gracePeriod=30 Oct 30 02:20:07.235164 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:07.235126 6221 generic.go:356] "Generic (PLEG): container finished" podID="24f7ec30-91bf-4af8-a913-f1650e340227" containerID="848d0f4e8d712e821cbfb0152c347aa883bff13b8400939a4da334a8219e2c62" exitCode=0 Oct 30 02:20:07.235364 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:07.235195 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a3a2ac-8456757f5d-8xzjx" event={"ID":"24f7ec30-91bf-4af8-a913-f1650e340227","Type":"ContainerDied","Data":"848d0f4e8d712e821cbfb0152c347aa883bff13b8400939a4da334a8219e2c62"} Oct 30 02:20:07.270220 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:07.270156 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a3a2ac-8456757f5d-8xzjx" Oct 30 02:20:07.401945 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:07.401908 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24f7ec30-91bf-4af8-a913-f1650e340227-kube-api-access-cg6d8\" (UniqueName: \"kubernetes.io/projected/24f7ec30-91bf-4af8-a913-f1650e340227-kube-api-access-cg6d8\") pod \"24f7ec30-91bf-4af8-a913-f1650e340227\" (UID: \"24f7ec30-91bf-4af8-a913-f1650e340227\") " Oct 30 02:20:07.402127 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:07.401964 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24f7ec30-91bf-4af8-a913-f1650e340227-metrics-certs\" (UniqueName: \"kubernetes.io/secret/24f7ec30-91bf-4af8-a913-f1650e340227-metrics-certs\") pod \"24f7ec30-91bf-4af8-a913-f1650e340227\" (UID: \"24f7ec30-91bf-4af8-a913-f1650e340227\") " Oct 30 02:20:07.402127 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:07.402102 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24f7ec30-91bf-4af8-a913-f1650e340227-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24f7ec30-91bf-4af8-a913-f1650e340227-maas-api-tls\") pod \"24f7ec30-91bf-4af8-a913-f1650e340227\" (UID: \"24f7ec30-91bf-4af8-a913-f1650e340227\") " Oct 30 02:20:07.402205 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:07.402154 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24f7ec30-91bf-4af8-a913-f1650e340227-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24f7ec30-91bf-4af8-a913-f1650e340227-openshift-service-ca\") pod \"24f7ec30-91bf-4af8-a913-f1650e340227\" (UID: \"24f7ec30-91bf-4af8-a913-f1650e340227\") " Oct 30 02:20:07.402621 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:07.402591 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24f7ec30-91bf-4af8-a913-f1650e340227-openshift-service-ca" pod "24f7ec30-91bf-4af8-a913-f1650e340227" (UID: "24f7ec30-91bf-4af8-a913-f1650e340227"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:20:07.404304 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:07.404276 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24f7ec30-91bf-4af8-a913-f1650e340227-maas-api-tls" pod "24f7ec30-91bf-4af8-a913-f1650e340227" (UID: "24f7ec30-91bf-4af8-a913-f1650e340227"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:20:07.404304 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:07.404288 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24f7ec30-91bf-4af8-a913-f1650e340227-metrics-certs" pod "24f7ec30-91bf-4af8-a913-f1650e340227" (UID: "24f7ec30-91bf-4af8-a913-f1650e340227"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:20:07.404452 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:07.404381 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24f7ec30-91bf-4af8-a913-f1650e340227-kube-api-access-cg6d8" pod "24f7ec30-91bf-4af8-a913-f1650e340227" (UID: "24f7ec30-91bf-4af8-a913-f1650e340227"). InnerVolumeSpecName "kube-api-access-cg6d8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:20:07.503339 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:07.503299 6221 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24f7ec30-91bf-4af8-a913-f1650e340227-maas-api-tls\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:20:07.503339 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:07.503331 6221 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24f7ec30-91bf-4af8-a913-f1650e340227-openshift-service-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:20:07.503339 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:07.503341 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg6d8\" (UniqueName: \"kubernetes.io/projected/24f7ec30-91bf-4af8-a913-f1650e340227-kube-api-access-cg6d8\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:20:07.503620 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:07.503354 6221 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24f7ec30-91bf-4af8-a913-f1650e340227-metrics-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:20:08.242867 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:08.242833 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a3a2ac-8456757f5d-8xzjx" Oct 30 02:20:08.243317 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:08.242867 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-a3a2ac-8456757f5d-8xzjx" event={"ID":"24f7ec30-91bf-4af8-a913-f1650e340227","Type":"ContainerDied","Data":"36294fff06d5566ae38b038b69aa3a072c0eafa69c008ae6841cf003ec6cf1a3"} Oct 30 02:20:08.243317 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:08.242933 6221 scope.go:122] "RemoveContainer" containerID="848d0f4e8d712e821cbfb0152c347aa883bff13b8400939a4da334a8219e2c62" Oct 30 02:20:08.295255 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:08.295223 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a3a2ac-8456757f5d-8xzjx"] Oct 30 02:20:08.307782 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:08.307745 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-a3a2ac-8456757f5d-8xzjx"] Oct 30 02:20:08.546497 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:08.546464 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24f7ec30-91bf-4af8-a913-f1650e340227" path="/var/lib/kubelet/pods/24f7ec30-91bf-4af8-a913-f1650e340227/volumes" Oct 30 02:20:08.546955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:08.546941 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa" path="/var/lib/kubelet/pods/666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa/volumes" Oct 30 02:20:11.150550 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:11.150491 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-a3a2ac-5d6ff74d96-dnpzn" podUID="702e03b2-6e41-4a27-837a-ede7f1c2e2c2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:20:11.150550 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:11.150508 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a3a2ac-6677b6f44c-bpm5m" podUID="26457d5d-6b8f-4e85-8bc1-321115791efc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:20:14.786648 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:14.786620 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-a3a2ac-5d6ff74d96-dnpzn" Oct 30 02:20:14.815387 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:14.815365 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a3a2ac-6677b6f44c-bpm5m" Oct 30 02:20:14.871493 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:14.871460 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/702e03b2-6e41-4a27-837a-ede7f1c2e2c2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/702e03b2-6e41-4a27-837a-ede7f1c2e2c2-plugins-config-volume\") pod \"702e03b2-6e41-4a27-837a-ede7f1c2e2c2\" (UID: \"702e03b2-6e41-4a27-837a-ede7f1c2e2c2\") " Oct 30 02:20:14.871493 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:14.871495 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26457d5d-6b8f-4e85-8bc1-321115791efc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26457d5d-6b8f-4e85-8bc1-321115791efc-plugins-config-volume\") pod \"26457d5d-6b8f-4e85-8bc1-321115791efc\" (UID: \"26457d5d-6b8f-4e85-8bc1-321115791efc\") " Oct 30 02:20:14.871788 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:14.871539 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/702e03b2-6e41-4a27-837a-ede7f1c2e2c2-kube-api-access-tlqdr\" (UniqueName: \"kubernetes.io/projected/702e03b2-6e41-4a27-837a-ede7f1c2e2c2-kube-api-access-tlqdr\") pod \"702e03b2-6e41-4a27-837a-ede7f1c2e2c2\" (UID: \"702e03b2-6e41-4a27-837a-ede7f1c2e2c2\") " Oct 30 02:20:14.871890 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:14.871865 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/702e03b2-6e41-4a27-837a-ede7f1c2e2c2-plugins-config-volume" pod "702e03b2-6e41-4a27-837a-ede7f1c2e2c2" (UID: "702e03b2-6e41-4a27-837a-ede7f1c2e2c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:20:14.871950 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:14.871908 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26457d5d-6b8f-4e85-8bc1-321115791efc-plugins-config-volume" pod "26457d5d-6b8f-4e85-8bc1-321115791efc" (UID: "26457d5d-6b8f-4e85-8bc1-321115791efc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:20:14.873851 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:14.873830 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/702e03b2-6e41-4a27-837a-ede7f1c2e2c2-kube-api-access-tlqdr" pod "702e03b2-6e41-4a27-837a-ede7f1c2e2c2" (UID: "702e03b2-6e41-4a27-837a-ede7f1c2e2c2"). InnerVolumeSpecName "kube-api-access-tlqdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:20:14.972471 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:14.972436 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26457d5d-6b8f-4e85-8bc1-321115791efc-kube-api-access-r8v7c\" (UniqueName: \"kubernetes.io/projected/26457d5d-6b8f-4e85-8bc1-321115791efc-kube-api-access-r8v7c\") pod \"26457d5d-6b8f-4e85-8bc1-321115791efc\" (UID: \"26457d5d-6b8f-4e85-8bc1-321115791efc\") " Oct 30 02:20:14.972722 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:14.972707 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/702e03b2-6e41-4a27-837a-ede7f1c2e2c2-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:20:14.972722 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:14.972723 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26457d5d-6b8f-4e85-8bc1-321115791efc-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:20:14.972853 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:14.972732 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlqdr\" (UniqueName: \"kubernetes.io/projected/702e03b2-6e41-4a27-837a-ede7f1c2e2c2-kube-api-access-tlqdr\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:20:14.976179 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:14.976147 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26457d5d-6b8f-4e85-8bc1-321115791efc-kube-api-access-r8v7c" pod "26457d5d-6b8f-4e85-8bc1-321115791efc" (UID: "26457d5d-6b8f-4e85-8bc1-321115791efc"). InnerVolumeSpecName "kube-api-access-r8v7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:20:15.073974 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:15.073935 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8v7c\" (UniqueName: \"kubernetes.io/projected/26457d5d-6b8f-4e85-8bc1-321115791efc-kube-api-access-r8v7c\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:20:15.283210 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:15.283161 6221 generic.go:356] "Generic (PLEG): container finished" podID="702e03b2-6e41-4a27-837a-ede7f1c2e2c2" containerID="dc649fe4c29d4979c29690108ae04f89428ace55d60d815535b1f72a8e0df35f" exitCode=0 Oct 30 02:20:15.283210 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:15.283212 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-a3a2ac-5d6ff74d96-dnpzn" Oct 30 02:20:15.283468 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:15.283252 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-a3a2ac-5d6ff74d96-dnpzn" event={"ID":"702e03b2-6e41-4a27-837a-ede7f1c2e2c2","Type":"ContainerDied","Data":"dc649fe4c29d4979c29690108ae04f89428ace55d60d815535b1f72a8e0df35f"} Oct 30 02:20:15.283468 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:15.283311 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-a3a2ac-5d6ff74d96-dnpzn" event={"ID":"702e03b2-6e41-4a27-837a-ede7f1c2e2c2","Type":"ContainerDied","Data":"e6ab79fc42e4b4458ff57383d381647a6d7e1c4420c53889fc75f8b4edd89be8"} Oct 30 02:20:15.283468 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:15.283343 6221 scope.go:122] "RemoveContainer" containerID="dc649fe4c29d4979c29690108ae04f89428ace55d60d815535b1f72a8e0df35f" Oct 30 02:20:15.284850 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:15.284824 6221 generic.go:356] "Generic (PLEG): container finished" podID="26457d5d-6b8f-4e85-8bc1-321115791efc" containerID="cf1461e454b5a0ca17822203ff258a06f43a988b751fd9de7cfae531a4ff22af" exitCode=0 Oct 30 02:20:15.285015 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:15.284883 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a3a2ac-6677b6f44c-bpm5m" Oct 30 02:20:15.285015 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:15.284905 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a3a2ac-6677b6f44c-bpm5m" event={"ID":"26457d5d-6b8f-4e85-8bc1-321115791efc","Type":"ContainerDied","Data":"cf1461e454b5a0ca17822203ff258a06f43a988b751fd9de7cfae531a4ff22af"} Oct 30 02:20:15.285015 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:15.284989 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a3a2ac-6677b6f44c-bpm5m" event={"ID":"26457d5d-6b8f-4e85-8bc1-321115791efc","Type":"ContainerDied","Data":"9941e92383ab3d0f214ee1dced98987fbcdd7a3ea916393b98be9649b5addb58"} Oct 30 02:20:15.294712 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:15.294685 6221 scope.go:122] "RemoveContainer" containerID="dc649fe4c29d4979c29690108ae04f89428ace55d60d815535b1f72a8e0df35f" Oct 30 02:20:15.295048 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:20:15.295023 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc649fe4c29d4979c29690108ae04f89428ace55d60d815535b1f72a8e0df35f\": container with ID starting with dc649fe4c29d4979c29690108ae04f89428ace55d60d815535b1f72a8e0df35f not found: ID does not exist" containerID="dc649fe4c29d4979c29690108ae04f89428ace55d60d815535b1f72a8e0df35f" Oct 30 02:20:15.295140 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:15.295057 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc649fe4c29d4979c29690108ae04f89428ace55d60d815535b1f72a8e0df35f"} err="failed to get container status \"dc649fe4c29d4979c29690108ae04f89428ace55d60d815535b1f72a8e0df35f\": rpc error: code = NotFound desc = could not find container \"dc649fe4c29d4979c29690108ae04f89428ace55d60d815535b1f72a8e0df35f\": container with ID starting with dc649fe4c29d4979c29690108ae04f89428ace55d60d815535b1f72a8e0df35f not found: ID does not exist" Oct 30 02:20:15.295140 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:15.295075 6221 scope.go:122] "RemoveContainer" containerID="cf1461e454b5a0ca17822203ff258a06f43a988b751fd9de7cfae531a4ff22af" Oct 30 02:20:15.303482 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:15.303458 6221 scope.go:122] "RemoveContainer" containerID="cf1461e454b5a0ca17822203ff258a06f43a988b751fd9de7cfae531a4ff22af" Oct 30 02:20:15.303757 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:20:15.303739 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf1461e454b5a0ca17822203ff258a06f43a988b751fd9de7cfae531a4ff22af\": container with ID starting with cf1461e454b5a0ca17822203ff258a06f43a988b751fd9de7cfae531a4ff22af not found: ID does not exist" containerID="cf1461e454b5a0ca17822203ff258a06f43a988b751fd9de7cfae531a4ff22af" Oct 30 02:20:15.303848 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:15.303765 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf1461e454b5a0ca17822203ff258a06f43a988b751fd9de7cfae531a4ff22af"} err="failed to get container status \"cf1461e454b5a0ca17822203ff258a06f43a988b751fd9de7cfae531a4ff22af\": rpc error: code = NotFound desc = could not find container \"cf1461e454b5a0ca17822203ff258a06f43a988b751fd9de7cfae531a4ff22af\": container with ID starting with cf1461e454b5a0ca17822203ff258a06f43a988b751fd9de7cfae531a4ff22af not found: ID does not exist" Oct 30 02:20:15.357399 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:15.357319 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a3a2ac-5d6ff74d96-dnpzn"] Oct 30 02:20:15.369485 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:15.369447 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-a3a2ac-5d6ff74d96-dnpzn"] Oct 30 02:20:15.408103 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:15.408074 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a3a2ac-6677b6f44c-bpm5m"] Oct 30 02:20:15.411522 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:15.411496 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a3a2ac-6677b6f44c-bpm5m"] Oct 30 02:20:16.548185 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:16.548150 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26457d5d-6b8f-4e85-8bc1-321115791efc" path="/var/lib/kubelet/pods/26457d5d-6b8f-4e85-8bc1-321115791efc/volumes" Oct 30 02:20:16.548605 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:16.548557 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="702e03b2-6e41-4a27-837a-ede7f1c2e2c2" path="/var/lib/kubelet/pods/702e03b2-6e41-4a27-837a-ede7f1c2e2c2/volumes" Oct 30 02:20:17.475510 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:17.475477 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-92f0cc-7865694dd6-98f7j"] Oct 30 02:20:17.504952 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:17.504920 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-92f0cc-74f74b5d79-whvzj"] Oct 30 02:20:19.638626 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:19.638586 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 02:20:21.494779 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:20:21.494736 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 02:20:22.476330 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:22.476290 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-92f0cc-7865694dd6-98f7j" podUID="f5083acb-200e-4c21-8879-b7a1ebc8d337" containerName="payload-processing" containerID="cri-o://943ba0264d375885b278adba7da6ec92fd1700718522e9c1e51136cd4c8a35b0" gracePeriod=25 Oct 30 02:20:22.505667 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:22.505627 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-92f0cc-74f74b5d79-whvzj" podUID="df8ed3ef-12e5-40a7-9322-4b2a5f921d60" containerName="payload-pre-processing" containerID="cri-o://cf562c45bc3367cc5b5b9f65ba0179d308f863c140d26f9308800fa0440f4008" gracePeriod=25 Oct 30 02:20:28.976119 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:28.976077 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-92f0cc-7865694dd6-98f7j" podUID="f5083acb-200e-4c21-8879-b7a1ebc8d337" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:20:28.976581 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:28.976117 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-92f0cc-74f74b5d79-whvzj" podUID="df8ed3ef-12e5-40a7-9322-4b2a5f921d60" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:20:32.358398 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:32.358356 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 02:20:38.976068 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:38.976027 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-92f0cc-74f74b5d79-whvzj" podUID="df8ed3ef-12e5-40a7-9322-4b2a5f921d60" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:20:38.976068 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:38.976050 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-92f0cc-7865694dd6-98f7j" podUID="f5083acb-200e-4c21-8879-b7a1ebc8d337" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:20:40.935857 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:40.935822 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-f1edaf-6947c677d8-fgcms"] Oct 30 02:20:42.756343 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:42.756318 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-92f0cc-7865694dd6-98f7j" Oct 30 02:20:42.782193 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:42.782167 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-92f0cc-74f74b5d79-whvzj" Oct 30 02:20:42.884480 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:42.884447 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df8ed3ef-12e5-40a7-9322-4b2a5f921d60-kube-api-access-89nwx\" (UniqueName: \"kubernetes.io/projected/df8ed3ef-12e5-40a7-9322-4b2a5f921d60-kube-api-access-89nwx\") pod \"df8ed3ef-12e5-40a7-9322-4b2a5f921d60\" (UID: \"df8ed3ef-12e5-40a7-9322-4b2a5f921d60\") " Oct 30 02:20:42.884480 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:42.884480 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5083acb-200e-4c21-8879-b7a1ebc8d337-kube-api-access-68jl9\" (UniqueName: \"kubernetes.io/projected/f5083acb-200e-4c21-8879-b7a1ebc8d337-kube-api-access-68jl9\") pod \"f5083acb-200e-4c21-8879-b7a1ebc8d337\" (UID: \"f5083acb-200e-4c21-8879-b7a1ebc8d337\") " Oct 30 02:20:42.884760 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:42.884516 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5083acb-200e-4c21-8879-b7a1ebc8d337-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5083acb-200e-4c21-8879-b7a1ebc8d337-plugins-config-volume\") pod \"f5083acb-200e-4c21-8879-b7a1ebc8d337\" (UID: \"f5083acb-200e-4c21-8879-b7a1ebc8d337\") " Oct 30 02:20:42.884760 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:42.884656 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df8ed3ef-12e5-40a7-9322-4b2a5f921d60-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df8ed3ef-12e5-40a7-9322-4b2a5f921d60-plugins-config-volume\") pod \"df8ed3ef-12e5-40a7-9322-4b2a5f921d60\" (UID: \"df8ed3ef-12e5-40a7-9322-4b2a5f921d60\") " Oct 30 02:20:42.885038 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:42.885004 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5083acb-200e-4c21-8879-b7a1ebc8d337-plugins-config-volume" pod "f5083acb-200e-4c21-8879-b7a1ebc8d337" (UID: "f5083acb-200e-4c21-8879-b7a1ebc8d337"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:20:42.885038 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:42.885025 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df8ed3ef-12e5-40a7-9322-4b2a5f921d60-plugins-config-volume" pod "df8ed3ef-12e5-40a7-9322-4b2a5f921d60" (UID: "df8ed3ef-12e5-40a7-9322-4b2a5f921d60"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:20:42.886716 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:42.886690 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5083acb-200e-4c21-8879-b7a1ebc8d337-kube-api-access-68jl9" pod "f5083acb-200e-4c21-8879-b7a1ebc8d337" (UID: "f5083acb-200e-4c21-8879-b7a1ebc8d337"). InnerVolumeSpecName "kube-api-access-68jl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:20:42.886837 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:42.886794 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df8ed3ef-12e5-40a7-9322-4b2a5f921d60-kube-api-access-89nwx" pod "df8ed3ef-12e5-40a7-9322-4b2a5f921d60" (UID: "df8ed3ef-12e5-40a7-9322-4b2a5f921d60"). InnerVolumeSpecName "kube-api-access-89nwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:20:42.985769 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:42.985732 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89nwx\" (UniqueName: \"kubernetes.io/projected/df8ed3ef-12e5-40a7-9322-4b2a5f921d60-kube-api-access-89nwx\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:20:42.985769 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:42.985762 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68jl9\" (UniqueName: \"kubernetes.io/projected/f5083acb-200e-4c21-8879-b7a1ebc8d337-kube-api-access-68jl9\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:20:42.985769 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:42.985771 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5083acb-200e-4c21-8879-b7a1ebc8d337-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:20:42.986024 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:42.985781 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df8ed3ef-12e5-40a7-9322-4b2a5f921d60-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:20:43.426641 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:43.426594 6221 generic.go:356] "Generic (PLEG): container finished" podID="f5083acb-200e-4c21-8879-b7a1ebc8d337" containerID="943ba0264d375885b278adba7da6ec92fd1700718522e9c1e51136cd4c8a35b0" exitCode=0 Oct 30 02:20:43.426641 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:43.426641 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-92f0cc-7865694dd6-98f7j" Oct 30 02:20:43.426920 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:43.426664 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-92f0cc-7865694dd6-98f7j" event={"ID":"f5083acb-200e-4c21-8879-b7a1ebc8d337","Type":"ContainerDied","Data":"943ba0264d375885b278adba7da6ec92fd1700718522e9c1e51136cd4c8a35b0"} Oct 30 02:20:43.426920 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:43.426724 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-92f0cc-7865694dd6-98f7j" event={"ID":"f5083acb-200e-4c21-8879-b7a1ebc8d337","Type":"ContainerDied","Data":"60b5061e48445bdc567227eca7db0cb0a8cbbfa70f4d6d46e44b397bd0baa6d4"} Oct 30 02:20:43.426920 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:43.426753 6221 scope.go:122] "RemoveContainer" containerID="943ba0264d375885b278adba7da6ec92fd1700718522e9c1e51136cd4c8a35b0" Oct 30 02:20:43.428297 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:43.428273 6221 generic.go:356] "Generic (PLEG): container finished" podID="df8ed3ef-12e5-40a7-9322-4b2a5f921d60" containerID="cf562c45bc3367cc5b5b9f65ba0179d308f863c140d26f9308800fa0440f4008" exitCode=0 Oct 30 02:20:43.428394 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:43.428322 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-92f0cc-74f74b5d79-whvzj" Oct 30 02:20:43.428394 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:43.428330 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-92f0cc-74f74b5d79-whvzj" event={"ID":"df8ed3ef-12e5-40a7-9322-4b2a5f921d60","Type":"ContainerDied","Data":"cf562c45bc3367cc5b5b9f65ba0179d308f863c140d26f9308800fa0440f4008"} Oct 30 02:20:43.428394 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:43.428373 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-92f0cc-74f74b5d79-whvzj" event={"ID":"df8ed3ef-12e5-40a7-9322-4b2a5f921d60","Type":"ContainerDied","Data":"b6c8f7cb0b3d0694b709c36f84e43f9cfc821dbcdff550ef1ede9ad6d82a88bc"} Oct 30 02:20:43.436905 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:43.436879 6221 scope.go:122] "RemoveContainer" containerID="943ba0264d375885b278adba7da6ec92fd1700718522e9c1e51136cd4c8a35b0" Oct 30 02:20:43.437164 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:20:43.437139 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"943ba0264d375885b278adba7da6ec92fd1700718522e9c1e51136cd4c8a35b0\": container with ID starting with 943ba0264d375885b278adba7da6ec92fd1700718522e9c1e51136cd4c8a35b0 not found: ID does not exist" containerID="943ba0264d375885b278adba7da6ec92fd1700718522e9c1e51136cd4c8a35b0" Oct 30 02:20:43.437249 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:43.437172 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"943ba0264d375885b278adba7da6ec92fd1700718522e9c1e51136cd4c8a35b0"} err="failed to get container status \"943ba0264d375885b278adba7da6ec92fd1700718522e9c1e51136cd4c8a35b0\": rpc error: code = NotFound desc = could not find container \"943ba0264d375885b278adba7da6ec92fd1700718522e9c1e51136cd4c8a35b0\": container with ID starting with 943ba0264d375885b278adba7da6ec92fd1700718522e9c1e51136cd4c8a35b0 not found: ID does not exist" Oct 30 02:20:43.437249 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:43.437188 6221 scope.go:122] "RemoveContainer" containerID="cf562c45bc3367cc5b5b9f65ba0179d308f863c140d26f9308800fa0440f4008" Oct 30 02:20:43.448727 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:43.448703 6221 scope.go:122] "RemoveContainer" containerID="cf562c45bc3367cc5b5b9f65ba0179d308f863c140d26f9308800fa0440f4008" Oct 30 02:20:43.448992 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:20:43.448975 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf562c45bc3367cc5b5b9f65ba0179d308f863c140d26f9308800fa0440f4008\": container with ID starting with cf562c45bc3367cc5b5b9f65ba0179d308f863c140d26f9308800fa0440f4008 not found: ID does not exist" containerID="cf562c45bc3367cc5b5b9f65ba0179d308f863c140d26f9308800fa0440f4008" Oct 30 02:20:43.449055 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:43.449000 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf562c45bc3367cc5b5b9f65ba0179d308f863c140d26f9308800fa0440f4008"} err="failed to get container status \"cf562c45bc3367cc5b5b9f65ba0179d308f863c140d26f9308800fa0440f4008\": rpc error: code = NotFound desc = could not find container \"cf562c45bc3367cc5b5b9f65ba0179d308f863c140d26f9308800fa0440f4008\": container with ID starting with cf562c45bc3367cc5b5b9f65ba0179d308f863c140d26f9308800fa0440f4008 not found: ID does not exist" Oct 30 02:20:43.479401 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:43.479362 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-92f0cc-74f74b5d79-whvzj"] Oct 30 02:20:43.491585 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:43.491554 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-92f0cc-74f74b5d79-whvzj"] Oct 30 02:20:43.525600 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:43.525566 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-92f0cc-7865694dd6-98f7j"] Oct 30 02:20:43.529021 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:43.528995 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-92f0cc-7865694dd6-98f7j"] Oct 30 02:20:44.547482 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:44.547449 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df8ed3ef-12e5-40a7-9322-4b2a5f921d60" path="/var/lib/kubelet/pods/df8ed3ef-12e5-40a7-9322-4b2a5f921d60/volumes" Oct 30 02:20:44.547912 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:44.547871 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5083acb-200e-4c21-8879-b7a1ebc8d337" path="/var/lib/kubelet/pods/f5083acb-200e-4c21-8879-b7a1ebc8d337/volumes" Oct 30 02:20:45.936420 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:45.936361 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-f1edaf-6947c677d8-fgcms" podUID="b124e8c4-1d49-4407-8841-99a368505a35" containerName="payload-processing" containerID="cri-o://6afe6533446cc041db65115a0f3d7d4bd187627015f3e4a82093426015466b2a" gracePeriod=25 Oct 30 02:20:49.719994 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:49.719846 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 02:20:51.493088 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:20:51.493051 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 02:20:51.874059 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:20:51.874020 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-f1edaf-6947c677d8-fgcms" podUID="b124e8c4-1d49-4407-8841-99a368505a35" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:21:01.873874 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:01.873828 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-f1edaf-6947c677d8-fgcms" podUID="b124e8c4-1d49-4407-8841-99a368505a35" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:21:04.606906 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:04.606870 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-abb30f-549b6b6b86-lsn24"] Oct 30 02:21:04.640872 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:04.640837 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-abb30f-dfb544599-pjbvp"] Oct 30 02:21:06.189062 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:06.189035 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-f1edaf-6947c677d8-fgcms" Oct 30 02:21:06.326729 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:06.326692 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b124e8c4-1d49-4407-8841-99a368505a35-kube-api-access-ll54v\" (UniqueName: \"kubernetes.io/projected/b124e8c4-1d49-4407-8841-99a368505a35-kube-api-access-ll54v\") pod \"b124e8c4-1d49-4407-8841-99a368505a35\" (UID: \"b124e8c4-1d49-4407-8841-99a368505a35\") " Oct 30 02:21:06.326947 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:06.326777 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b124e8c4-1d49-4407-8841-99a368505a35-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b124e8c4-1d49-4407-8841-99a368505a35-plugins-config-volume\") pod \"b124e8c4-1d49-4407-8841-99a368505a35\" (UID: \"b124e8c4-1d49-4407-8841-99a368505a35\") " Oct 30 02:21:06.327162 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:06.327136 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b124e8c4-1d49-4407-8841-99a368505a35-plugins-config-volume" pod "b124e8c4-1d49-4407-8841-99a368505a35" (UID: "b124e8c4-1d49-4407-8841-99a368505a35"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:21:06.328893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:06.328868 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b124e8c4-1d49-4407-8841-99a368505a35-kube-api-access-ll54v" pod "b124e8c4-1d49-4407-8841-99a368505a35" (UID: "b124e8c4-1d49-4407-8841-99a368505a35"). InnerVolumeSpecName "kube-api-access-ll54v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:21:06.427870 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:06.427834 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll54v\" (UniqueName: \"kubernetes.io/projected/b124e8c4-1d49-4407-8841-99a368505a35-kube-api-access-ll54v\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:21:06.427870 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:06.427867 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b124e8c4-1d49-4407-8841-99a368505a35-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:21:06.557148 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:06.557117 6221 generic.go:356] "Generic (PLEG): container finished" podID="b124e8c4-1d49-4407-8841-99a368505a35" containerID="6afe6533446cc041db65115a0f3d7d4bd187627015f3e4a82093426015466b2a" exitCode=0 Oct 30 02:21:06.557314 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:06.557163 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-f1edaf-6947c677d8-fgcms" Oct 30 02:21:06.557314 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:06.557196 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-f1edaf-6947c677d8-fgcms" event={"ID":"b124e8c4-1d49-4407-8841-99a368505a35","Type":"ContainerDied","Data":"6afe6533446cc041db65115a0f3d7d4bd187627015f3e4a82093426015466b2a"} Oct 30 02:21:06.557314 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:06.557248 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-f1edaf-6947c677d8-fgcms" event={"ID":"b124e8c4-1d49-4407-8841-99a368505a35","Type":"ContainerDied","Data":"9c4abd7825472723cf55b7647041f573d310e64286b0e50f1035d41337e8e63b"} Oct 30 02:21:06.557314 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:06.557270 6221 scope.go:122] "RemoveContainer" containerID="6afe6533446cc041db65115a0f3d7d4bd187627015f3e4a82093426015466b2a" Oct 30 02:21:06.565821 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:06.565798 6221 scope.go:122] "RemoveContainer" containerID="6afe6533446cc041db65115a0f3d7d4bd187627015f3e4a82093426015466b2a" Oct 30 02:21:06.566104 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:21:06.566083 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6afe6533446cc041db65115a0f3d7d4bd187627015f3e4a82093426015466b2a\": container with ID starting with 6afe6533446cc041db65115a0f3d7d4bd187627015f3e4a82093426015466b2a not found: ID does not exist" containerID="6afe6533446cc041db65115a0f3d7d4bd187627015f3e4a82093426015466b2a" Oct 30 02:21:06.566169 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:06.566117 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6afe6533446cc041db65115a0f3d7d4bd187627015f3e4a82093426015466b2a"} err="failed to get container status \"6afe6533446cc041db65115a0f3d7d4bd187627015f3e4a82093426015466b2a\": rpc error: code = NotFound desc = could not find container \"6afe6533446cc041db65115a0f3d7d4bd187627015f3e4a82093426015466b2a\": container with ID starting with 6afe6533446cc041db65115a0f3d7d4bd187627015f3e4a82093426015466b2a not found: ID does not exist" Oct 30 02:21:06.608264 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:06.608194 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-f1edaf-6947c677d8-fgcms"] Oct 30 02:21:06.620645 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:06.620620 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-f1edaf-6947c677d8-fgcms"] Oct 30 02:21:08.550384 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:08.550296 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b124e8c4-1d49-4407-8841-99a368505a35" path="/var/lib/kubelet/pods/b124e8c4-1d49-4407-8841-99a368505a35/volumes" Oct 30 02:21:09.607550 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:09.607486 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-abb30f-549b6b6b86-lsn24" podUID="a577ca5d-a4d6-4f5c-8f0e-422290e6ec61" containerName="payload-processing" containerID="cri-o://8116d6c60940e8edb4dab1ab98cdf66941cfb3f7cd0599a1d45347737f6a09de" gracePeriod=25 Oct 30 02:21:09.642047 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:09.641966 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-abb30f-dfb544599-pjbvp" podUID="fcbfae96-e276-4753-acb5-59ee55640aa8" containerName="payload-pre-processing" containerID="cri-o://fdfa2efd1150de51a574ed459f115172655639f9771909ca0ceb8ddaf372fb67" gracePeriod=25 Oct 30 02:21:13.124379 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:13.124326 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 02:21:19.578650 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:19.578604 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-abb30f-549b6b6b86-lsn24" podUID="a577ca5d-a4d6-4f5c-8f0e-422290e6ec61" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:21:19.579130 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:19.578604 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-abb30f-dfb544599-pjbvp" podUID="fcbfae96-e276-4753-acb5-59ee55640aa8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:21:21.493571 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:21:21.493517 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 02:21:28.379083 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:28.379046 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-861cd2-6477f778c6-8cbg5"] Oct 30 02:21:29.578089 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:29.578049 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-abb30f-dfb544599-pjbvp" podUID="fcbfae96-e276-4753-acb5-59ee55640aa8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:21:29.578089 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:29.578078 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-abb30f-549b6b6b86-lsn24" podUID="a577ca5d-a4d6-4f5c-8f0e-422290e6ec61" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:21:29.884516 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:29.884470 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-abb30f-549b6b6b86-lsn24" Oct 30 02:21:29.942249 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:29.942226 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-abb30f-dfb544599-pjbvp" Oct 30 02:21:29.992179 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:29.992140 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcbfae96-e276-4753-acb5-59ee55640aa8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcbfae96-e276-4753-acb5-59ee55640aa8-plugins-config-volume\") pod \"fcbfae96-e276-4753-acb5-59ee55640aa8\" (UID: \"fcbfae96-e276-4753-acb5-59ee55640aa8\") " Oct 30 02:21:29.992379 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:29.992281 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcbfae96-e276-4753-acb5-59ee55640aa8-kube-api-access-s9mmf\" (UniqueName: \"kubernetes.io/projected/fcbfae96-e276-4753-acb5-59ee55640aa8-kube-api-access-s9mmf\") pod \"fcbfae96-e276-4753-acb5-59ee55640aa8\" (UID: \"fcbfae96-e276-4753-acb5-59ee55640aa8\") " Oct 30 02:21:29.992379 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:29.992305 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a577ca5d-a4d6-4f5c-8f0e-422290e6ec61-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a577ca5d-a4d6-4f5c-8f0e-422290e6ec61-plugins-config-volume\") pod \"a577ca5d-a4d6-4f5c-8f0e-422290e6ec61\" (UID: \"a577ca5d-a4d6-4f5c-8f0e-422290e6ec61\") " Oct 30 02:21:29.992379 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:29.992325 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a577ca5d-a4d6-4f5c-8f0e-422290e6ec61-kube-api-access-zb57d\" (UniqueName: \"kubernetes.io/projected/a577ca5d-a4d6-4f5c-8f0e-422290e6ec61-kube-api-access-zb57d\") pod \"a577ca5d-a4d6-4f5c-8f0e-422290e6ec61\" (UID: \"a577ca5d-a4d6-4f5c-8f0e-422290e6ec61\") " Oct 30 02:21:29.992707 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:29.992681 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcbfae96-e276-4753-acb5-59ee55640aa8-plugins-config-volume" pod "fcbfae96-e276-4753-acb5-59ee55640aa8" (UID: "fcbfae96-e276-4753-acb5-59ee55640aa8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:21:29.992787 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:29.992720 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a577ca5d-a4d6-4f5c-8f0e-422290e6ec61-plugins-config-volume" pod "a577ca5d-a4d6-4f5c-8f0e-422290e6ec61" (UID: "a577ca5d-a4d6-4f5c-8f0e-422290e6ec61"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:21:29.994562 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:29.994514 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a577ca5d-a4d6-4f5c-8f0e-422290e6ec61-kube-api-access-zb57d" pod "a577ca5d-a4d6-4f5c-8f0e-422290e6ec61" (UID: "a577ca5d-a4d6-4f5c-8f0e-422290e6ec61"). InnerVolumeSpecName "kube-api-access-zb57d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:21:29.994748 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:29.994718 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcbfae96-e276-4753-acb5-59ee55640aa8-kube-api-access-s9mmf" pod "fcbfae96-e276-4753-acb5-59ee55640aa8" (UID: "fcbfae96-e276-4753-acb5-59ee55640aa8"). InnerVolumeSpecName "kube-api-access-s9mmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:21:30.094064 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:30.094030 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9mmf\" (UniqueName: \"kubernetes.io/projected/fcbfae96-e276-4753-acb5-59ee55640aa8-kube-api-access-s9mmf\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:21:30.094064 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:30.094059 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a577ca5d-a4d6-4f5c-8f0e-422290e6ec61-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:21:30.094064 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:30.094069 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb57d\" (UniqueName: \"kubernetes.io/projected/a577ca5d-a4d6-4f5c-8f0e-422290e6ec61-kube-api-access-zb57d\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:21:30.094315 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:30.094079 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fcbfae96-e276-4753-acb5-59ee55640aa8-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:21:30.671301 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:30.671261 6221 generic.go:356] "Generic (PLEG): container finished" podID="fcbfae96-e276-4753-acb5-59ee55640aa8" containerID="fdfa2efd1150de51a574ed459f115172655639f9771909ca0ceb8ddaf372fb67" exitCode=0 Oct 30 02:21:30.671857 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:30.671329 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-abb30f-dfb544599-pjbvp" Oct 30 02:21:30.671857 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:30.671330 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-abb30f-dfb544599-pjbvp" event={"ID":"fcbfae96-e276-4753-acb5-59ee55640aa8","Type":"ContainerDied","Data":"fdfa2efd1150de51a574ed459f115172655639f9771909ca0ceb8ddaf372fb67"} Oct 30 02:21:30.671857 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:30.671478 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-abb30f-dfb544599-pjbvp" event={"ID":"fcbfae96-e276-4753-acb5-59ee55640aa8","Type":"ContainerDied","Data":"3cda9e6ad6eda8b53adbc7d94f677d72479455c80b069d5f7c8475b70688981d"} Oct 30 02:21:30.671857 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:30.671519 6221 scope.go:122] "RemoveContainer" containerID="fdfa2efd1150de51a574ed459f115172655639f9771909ca0ceb8ddaf372fb67" Oct 30 02:21:30.672750 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:30.672626 6221 generic.go:356] "Generic (PLEG): container finished" podID="a577ca5d-a4d6-4f5c-8f0e-422290e6ec61" containerID="8116d6c60940e8edb4dab1ab98cdf66941cfb3f7cd0599a1d45347737f6a09de" exitCode=0 Oct 30 02:21:30.672750 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:30.672655 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-abb30f-549b6b6b86-lsn24" event={"ID":"a577ca5d-a4d6-4f5c-8f0e-422290e6ec61","Type":"ContainerDied","Data":"8116d6c60940e8edb4dab1ab98cdf66941cfb3f7cd0599a1d45347737f6a09de"} Oct 30 02:21:30.672750 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:30.672682 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-abb30f-549b6b6b86-lsn24" event={"ID":"a577ca5d-a4d6-4f5c-8f0e-422290e6ec61","Type":"ContainerDied","Data":"eb152ae76d4239d668fd7080e30562d3adf71cbae4cde78d3babf155bfbf995b"} Oct 30 02:21:30.672750 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:30.672684 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-abb30f-549b6b6b86-lsn24" Oct 30 02:21:30.680425 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:30.680405 6221 scope.go:122] "RemoveContainer" containerID="fdfa2efd1150de51a574ed459f115172655639f9771909ca0ceb8ddaf372fb67" Oct 30 02:21:30.680702 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:21:30.680682 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdfa2efd1150de51a574ed459f115172655639f9771909ca0ceb8ddaf372fb67\": container with ID starting with fdfa2efd1150de51a574ed459f115172655639f9771909ca0ceb8ddaf372fb67 not found: ID does not exist" containerID="fdfa2efd1150de51a574ed459f115172655639f9771909ca0ceb8ddaf372fb67" Oct 30 02:21:30.680765 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:30.680710 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdfa2efd1150de51a574ed459f115172655639f9771909ca0ceb8ddaf372fb67"} err="failed to get container status \"fdfa2efd1150de51a574ed459f115172655639f9771909ca0ceb8ddaf372fb67\": rpc error: code = NotFound desc = could not find container \"fdfa2efd1150de51a574ed459f115172655639f9771909ca0ceb8ddaf372fb67\": container with ID starting with fdfa2efd1150de51a574ed459f115172655639f9771909ca0ceb8ddaf372fb67 not found: ID does not exist" Oct 30 02:21:30.680765 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:30.680727 6221 scope.go:122] "RemoveContainer" containerID="8116d6c60940e8edb4dab1ab98cdf66941cfb3f7cd0599a1d45347737f6a09de" Oct 30 02:21:30.688708 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:30.688690 6221 scope.go:122] "RemoveContainer" containerID="8116d6c60940e8edb4dab1ab98cdf66941cfb3f7cd0599a1d45347737f6a09de" Oct 30 02:21:30.689006 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:21:30.688985 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8116d6c60940e8edb4dab1ab98cdf66941cfb3f7cd0599a1d45347737f6a09de\": container with ID starting with 8116d6c60940e8edb4dab1ab98cdf66941cfb3f7cd0599a1d45347737f6a09de not found: ID does not exist" containerID="8116d6c60940e8edb4dab1ab98cdf66941cfb3f7cd0599a1d45347737f6a09de" Oct 30 02:21:30.689070 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:30.689013 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8116d6c60940e8edb4dab1ab98cdf66941cfb3f7cd0599a1d45347737f6a09de"} err="failed to get container status \"8116d6c60940e8edb4dab1ab98cdf66941cfb3f7cd0599a1d45347737f6a09de\": rpc error: code = NotFound desc = could not find container \"8116d6c60940e8edb4dab1ab98cdf66941cfb3f7cd0599a1d45347737f6a09de\": container with ID starting with 8116d6c60940e8edb4dab1ab98cdf66941cfb3f7cd0599a1d45347737f6a09de not found: ID does not exist" Oct 30 02:21:30.740171 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:30.740134 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-abb30f-dfb544599-pjbvp"] Oct 30 02:21:30.752632 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:30.752598 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-abb30f-dfb544599-pjbvp"] Oct 30 02:21:30.796871 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:30.796835 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-abb30f-549b6b6b86-lsn24"] Oct 30 02:21:30.807931 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:30.807902 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-abb30f-549b6b6b86-lsn24"] Oct 30 02:21:32.014347 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:32.014307 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 02:21:32.546210 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:32.546175 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a577ca5d-a4d6-4f5c-8f0e-422290e6ec61" path="/var/lib/kubelet/pods/a577ca5d-a4d6-4f5c-8f0e-422290e6ec61/volumes" Oct 30 02:21:32.546627 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:32.546612 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcbfae96-e276-4753-acb5-59ee55640aa8" path="/var/lib/kubelet/pods/fcbfae96-e276-4753-acb5-59ee55640aa8/volumes" Oct 30 02:21:33.380075 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:33.380024 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-861cd2-6477f778c6-8cbg5" podUID="e4160a65-33dc-4c24-8a6d-f24c37cb3ebe" containerName="payload-processing" containerID="cri-o://78f3e7ed3ccaf97a4e8ee4df3cb62753455c143dd03264f121ea112d0763ce42" gracePeriod=25 Oct 30 02:21:35.332032 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:35.331988 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-861cd2-6477f778c6-8cbg5" podUID="e4160a65-33dc-4c24-8a6d-f24c37cb3ebe" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:21:37.124566 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:37.111351 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp"] Oct 30 02:21:37.124566 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:37.112591 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" podUID="51dc5d9e-31aa-4246-9bd6-8953c55424f1" containerName="istio-proxy" containerID="cri-o://453e8271f6f7cc307c88daa03eff05c7b1047b9e97e07910e9b740c235b58d7e" gracePeriod=30 Oct 30 02:21:42.373612 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:42.373583 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" Oct 30 02:21:42.520238 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:42.520157 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51dc5d9e-31aa-4246-9bd6-8953c55424f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51dc5d9e-31aa-4246-9bd6-8953c55424f1-istio-envoy\") pod \"51dc5d9e-31aa-4246-9bd6-8953c55424f1\" (UID: \"51dc5d9e-31aa-4246-9bd6-8953c55424f1\") " Oct 30 02:21:42.520238 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:42.520207 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51dc5d9e-31aa-4246-9bd6-8953c55424f1-kube-api-access-zjtkj\" (UniqueName: \"kubernetes.io/projected/51dc5d9e-31aa-4246-9bd6-8953c55424f1-kube-api-access-zjtkj\") pod \"51dc5d9e-31aa-4246-9bd6-8953c55424f1\" (UID: \"51dc5d9e-31aa-4246-9bd6-8953c55424f1\") " Oct 30 02:21:42.520473 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:42.520274 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51dc5d9e-31aa-4246-9bd6-8953c55424f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51dc5d9e-31aa-4246-9bd6-8953c55424f1-istio-data\") pod \"51dc5d9e-31aa-4246-9bd6-8953c55424f1\" (UID: \"51dc5d9e-31aa-4246-9bd6-8953c55424f1\") " Oct 30 02:21:42.520473 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:42.520299 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51dc5d9e-31aa-4246-9bd6-8953c55424f1-istio-token\" (UniqueName: \"kubernetes.io/projected/51dc5d9e-31aa-4246-9bd6-8953c55424f1-istio-token\") pod \"51dc5d9e-31aa-4246-9bd6-8953c55424f1\" (UID: \"51dc5d9e-31aa-4246-9bd6-8953c55424f1\") " Oct 30 02:21:42.520473 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:42.520323 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51dc5d9e-31aa-4246-9bd6-8953c55424f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51dc5d9e-31aa-4246-9bd6-8953c55424f1-credential-socket\") pod \"51dc5d9e-31aa-4246-9bd6-8953c55424f1\" (UID: \"51dc5d9e-31aa-4246-9bd6-8953c55424f1\") " Oct 30 02:21:42.520473 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:42.520341 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51dc5d9e-31aa-4246-9bd6-8953c55424f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51dc5d9e-31aa-4246-9bd6-8953c55424f1-istiod-ca-cert\") pod \"51dc5d9e-31aa-4246-9bd6-8953c55424f1\" (UID: \"51dc5d9e-31aa-4246-9bd6-8953c55424f1\") " Oct 30 02:21:42.520473 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:42.520369 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51dc5d9e-31aa-4246-9bd6-8953c55424f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51dc5d9e-31aa-4246-9bd6-8953c55424f1-workload-certs\") pod \"51dc5d9e-31aa-4246-9bd6-8953c55424f1\" (UID: \"51dc5d9e-31aa-4246-9bd6-8953c55424f1\") " Oct 30 02:21:42.520473 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:42.520445 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51dc5d9e-31aa-4246-9bd6-8953c55424f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51dc5d9e-31aa-4246-9bd6-8953c55424f1-workload-socket\") pod \"51dc5d9e-31aa-4246-9bd6-8953c55424f1\" (UID: \"51dc5d9e-31aa-4246-9bd6-8953c55424f1\") " Oct 30 02:21:42.520865 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:42.520480 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51dc5d9e-31aa-4246-9bd6-8953c55424f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51dc5d9e-31aa-4246-9bd6-8953c55424f1-istio-podinfo\") pod \"51dc5d9e-31aa-4246-9bd6-8953c55424f1\" (UID: \"51dc5d9e-31aa-4246-9bd6-8953c55424f1\") " Oct 30 02:21:42.520865 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:42.520643 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51dc5d9e-31aa-4246-9bd6-8953c55424f1-istio-data" pod "51dc5d9e-31aa-4246-9bd6-8953c55424f1" (UID: "51dc5d9e-31aa-4246-9bd6-8953c55424f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:21:42.520865 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:42.520822 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51dc5d9e-31aa-4246-9bd6-8953c55424f1-workload-certs" pod "51dc5d9e-31aa-4246-9bd6-8953c55424f1" (UID: "51dc5d9e-31aa-4246-9bd6-8953c55424f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:21:42.520865 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:42.520652 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51dc5d9e-31aa-4246-9bd6-8953c55424f1-credential-socket" pod "51dc5d9e-31aa-4246-9bd6-8953c55424f1" (UID: "51dc5d9e-31aa-4246-9bd6-8953c55424f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:21:42.521093 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:42.520868 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51dc5d9e-31aa-4246-9bd6-8953c55424f1-workload-socket" pod "51dc5d9e-31aa-4246-9bd6-8953c55424f1" (UID: "51dc5d9e-31aa-4246-9bd6-8953c55424f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:21:42.521093 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:42.520904 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51dc5d9e-31aa-4246-9bd6-8953c55424f1-istiod-ca-cert" pod "51dc5d9e-31aa-4246-9bd6-8953c55424f1" (UID: "51dc5d9e-31aa-4246-9bd6-8953c55424f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:21:42.521093 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:42.520949 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51dc5d9e-31aa-4246-9bd6-8953c55424f1-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:21:42.521093 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:42.520971 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51dc5d9e-31aa-4246-9bd6-8953c55424f1-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:21:42.521093 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:42.520987 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51dc5d9e-31aa-4246-9bd6-8953c55424f1-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:21:42.522591 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:42.522558 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51dc5d9e-31aa-4246-9bd6-8953c55424f1-istio-envoy" pod "51dc5d9e-31aa-4246-9bd6-8953c55424f1" (UID: "51dc5d9e-31aa-4246-9bd6-8953c55424f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:21:42.522717 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:42.522624 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51dc5d9e-31aa-4246-9bd6-8953c55424f1-istio-token" pod "51dc5d9e-31aa-4246-9bd6-8953c55424f1" (UID: "51dc5d9e-31aa-4246-9bd6-8953c55424f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:21:42.522717 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:42.522641 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51dc5d9e-31aa-4246-9bd6-8953c55424f1-istio-podinfo" pod "51dc5d9e-31aa-4246-9bd6-8953c55424f1" (UID: "51dc5d9e-31aa-4246-9bd6-8953c55424f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:21:42.522811 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:42.522761 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51dc5d9e-31aa-4246-9bd6-8953c55424f1-kube-api-access-zjtkj" pod "51dc5d9e-31aa-4246-9bd6-8953c55424f1" (UID: "51dc5d9e-31aa-4246-9bd6-8953c55424f1"). InnerVolumeSpecName "kube-api-access-zjtkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:21:42.619137 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:42.619099 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq"] Oct 30 02:21:42.622297 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:42.622271 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51dc5d9e-31aa-4246-9bd6-8953c55424f1-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:21:42.622297 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:42.622298 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51dc5d9e-31aa-4246-9bd6-8953c55424f1-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:21:42.622297 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:42.622310 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51dc5d9e-31aa-4246-9bd6-8953c55424f1-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:21:42.622663 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:42.622323 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjtkj\" (UniqueName: \"kubernetes.io/projected/51dc5d9e-31aa-4246-9bd6-8953c55424f1-kube-api-access-zjtkj\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:21:42.622663 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:42.622336 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51dc5d9e-31aa-4246-9bd6-8953c55424f1-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:21:42.622663 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:42.622349 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51dc5d9e-31aa-4246-9bd6-8953c55424f1-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:21:42.626022 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:42.625997 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t"] Oct 30 02:21:42.732604 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:42.732570 6221 generic.go:356] "Generic (PLEG): container finished" podID="51dc5d9e-31aa-4246-9bd6-8953c55424f1" containerID="453e8271f6f7cc307c88daa03eff05c7b1047b9e97e07910e9b740c235b58d7e" exitCode=0 Oct 30 02:21:42.732768 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:42.732614 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" event={"ID":"51dc5d9e-31aa-4246-9bd6-8953c55424f1","Type":"ContainerDied","Data":"453e8271f6f7cc307c88daa03eff05c7b1047b9e97e07910e9b740c235b58d7e"} Oct 30 02:21:42.732768 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:42.732624 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" Oct 30 02:21:42.732768 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:42.732686 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp" event={"ID":"51dc5d9e-31aa-4246-9bd6-8953c55424f1","Type":"ContainerDied","Data":"3ad2eaee5e00a4d627b321483b69aa83d4b4c2b1f734a5757709fd16e43e624a"} Oct 30 02:21:42.732768 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:42.732711 6221 scope.go:122] "RemoveContainer" containerID="453e8271f6f7cc307c88daa03eff05c7b1047b9e97e07910e9b740c235b58d7e" Oct 30 02:21:42.741649 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:42.741627 6221 scope.go:122] "RemoveContainer" containerID="43303036d7d0a45bfc51c6bd52645cd1b9c19a912ac61ad1fe416e6f4e6d3baa" Oct 30 02:21:42.749519 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:42.749499 6221 scope.go:122] "RemoveContainer" containerID="453e8271f6f7cc307c88daa03eff05c7b1047b9e97e07910e9b740c235b58d7e" Oct 30 02:21:42.749880 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:21:42.749860 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"453e8271f6f7cc307c88daa03eff05c7b1047b9e97e07910e9b740c235b58d7e\": container with ID starting with 453e8271f6f7cc307c88daa03eff05c7b1047b9e97e07910e9b740c235b58d7e not found: ID does not exist" containerID="453e8271f6f7cc307c88daa03eff05c7b1047b9e97e07910e9b740c235b58d7e" Oct 30 02:21:42.750013 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:42.749887 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"453e8271f6f7cc307c88daa03eff05c7b1047b9e97e07910e9b740c235b58d7e"} err="failed to get container status \"453e8271f6f7cc307c88daa03eff05c7b1047b9e97e07910e9b740c235b58d7e\": rpc error: code = NotFound desc = could not find container \"453e8271f6f7cc307c88daa03eff05c7b1047b9e97e07910e9b740c235b58d7e\": container with ID starting with 453e8271f6f7cc307c88daa03eff05c7b1047b9e97e07910e9b740c235b58d7e not found: ID does not exist" Oct 30 02:21:42.750013 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:42.749903 6221 scope.go:122] "RemoveContainer" containerID="43303036d7d0a45bfc51c6bd52645cd1b9c19a912ac61ad1fe416e6f4e6d3baa" Oct 30 02:21:42.750207 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:21:42.750185 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43303036d7d0a45bfc51c6bd52645cd1b9c19a912ac61ad1fe416e6f4e6d3baa\": container with ID starting with 43303036d7d0a45bfc51c6bd52645cd1b9c19a912ac61ad1fe416e6f4e6d3baa not found: ID does not exist" containerID="43303036d7d0a45bfc51c6bd52645cd1b9c19a912ac61ad1fe416e6f4e6d3baa" Oct 30 02:21:42.750304 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:42.750215 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43303036d7d0a45bfc51c6bd52645cd1b9c19a912ac61ad1fe416e6f4e6d3baa"} err="failed to get container status \"43303036d7d0a45bfc51c6bd52645cd1b9c19a912ac61ad1fe416e6f4e6d3baa\": rpc error: code = NotFound desc = could not find container \"43303036d7d0a45bfc51c6bd52645cd1b9c19a912ac61ad1fe416e6f4e6d3baa\": container with ID starting with 43303036d7d0a45bfc51c6bd52645cd1b9c19a912ac61ad1fe416e6f4e6d3baa not found: ID does not exist" Oct 30 02:21:42.791560 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:42.791519 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp"] Oct 30 02:21:42.804181 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:42.804159 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-861cd2-openshift-default-5c46dff555-blxkp"] Oct 30 02:21:44.546027 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:44.545995 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51dc5d9e-31aa-4246-9bd6-8953c55424f1" path="/var/lib/kubelet/pods/51dc5d9e-31aa-4246-9bd6-8953c55424f1/volumes" Oct 30 02:21:45.332288 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:45.332247 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-861cd2-6477f778c6-8cbg5" podUID="e4160a65-33dc-4c24-8a6d-f24c37cb3ebe" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 02:21:51.172774 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.172735 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07258647-dbt65"] Oct 30 02:21:51.173329 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173308 6221 cpu_manager.go:414] "containerMap: removing container" podUID="9061008c-04d0-42d9-96ae-a8ebe16c54b7" containerName="payload-pre-processing" Oct 30 02:21:51.173421 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173331 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9061008c-04d0-42d9-96ae-a8ebe16c54b7" containerName="payload-pre-processing" Oct 30 02:21:51.173421 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173354 6221 cpu_manager.go:414] "containerMap: removing container" podUID="a5227b83-d834-4276-bd61-27e1251a0708" containerName="istio-proxy" Oct 30 02:21:51.173421 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173362 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5227b83-d834-4276-bd61-27e1251a0708" containerName="istio-proxy" Oct 30 02:21:51.173421 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173374 6221 cpu_manager.go:414] "containerMap: removing container" podUID="c4ac5f29-bf87-42b2-82af-ee49cd16e879" containerName="cleanup" Oct 30 02:21:51.173421 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173382 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4ac5f29-bf87-42b2-82af-ee49cd16e879" containerName="cleanup" Oct 30 02:21:51.173421 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173392 6221 cpu_manager.go:414] "containerMap: removing container" podUID="d90c1ed4-5a28-4ce1-ba6e-c521698c3667" containerName="cleanup" Oct 30 02:21:51.173421 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173399 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d90c1ed4-5a28-4ce1-ba6e-c521698c3667" containerName="cleanup" Oct 30 02:21:51.173421 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173415 6221 cpu_manager.go:414] "containerMap: removing container" podUID="348f8c54-8ced-40e2-846d-f551e78b3092" containerName="maas-api" Oct 30 02:21:51.173421 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173423 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="348f8c54-8ced-40e2-846d-f551e78b3092" containerName="maas-api" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173434 6221 cpu_manager.go:414] "containerMap: removing container" podUID="6b685aa0-0c58-4911-8f31-3e920b8151a1" containerName="istio-proxy" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173444 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b685aa0-0c58-4911-8f31-3e920b8151a1" containerName="istio-proxy" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173456 6221 cpu_manager.go:414] "containerMap: removing container" podUID="16bdf204-496b-4e72-b5cd-e7b5719043e5" containerName="payload-processing" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173464 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16bdf204-496b-4e72-b5cd-e7b5719043e5" containerName="payload-processing" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173474 6221 cpu_manager.go:414] "containerMap: removing container" podUID="def2849e-c94c-4685-9992-e1121b744ac8" containerName="cleanup" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173489 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="def2849e-c94c-4685-9992-e1121b744ac8" containerName="cleanup" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173498 6221 cpu_manager.go:414] "containerMap: removing container" podUID="def2849e-c94c-4685-9992-e1121b744ac8" containerName="cleanup" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173505 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="def2849e-c94c-4685-9992-e1121b744ac8" containerName="cleanup" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173519 6221 cpu_manager.go:414] "containerMap: removing container" podUID="666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa" containerName="istio-proxy" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173544 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa" containerName="istio-proxy" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173554 6221 cpu_manager.go:414] "containerMap: removing container" podUID="e9566007-9280-43dc-8c1d-0439ccc4f679" containerName="payload-pre-processing" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173561 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9566007-9280-43dc-8c1d-0439ccc4f679" containerName="payload-pre-processing" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173571 6221 cpu_manager.go:414] "containerMap: removing container" podUID="27948676-86ab-49f1-a4f7-3083a591738c" containerName="payload-processing" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173577 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27948676-86ab-49f1-a4f7-3083a591738c" containerName="payload-processing" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173588 6221 cpu_manager.go:414] "containerMap: removing container" podUID="ff0dc863-4ea6-454f-9f61-5bfd9a9db849" containerName="istio-proxy" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173595 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff0dc863-4ea6-454f-9f61-5bfd9a9db849" containerName="istio-proxy" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173604 6221 cpu_manager.go:414] "containerMap: removing container" podUID="c26c4344-4ccb-44d9-ad1d-cfd8720bc56d" containerName="maas-api" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173611 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c26c4344-4ccb-44d9-ad1d-cfd8720bc56d" containerName="maas-api" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173624 6221 cpu_manager.go:414] "containerMap: removing container" podUID="018c3fe2-3e67-45ed-a3d7-d70f3fe5b956" containerName="main" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173631 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="018c3fe2-3e67-45ed-a3d7-d70f3fe5b956" containerName="main" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173647 6221 cpu_manager.go:414] "containerMap: removing container" podUID="7562db53-8060-4df5-a62d-699d3a44caa9" containerName="maas-api" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173656 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7562db53-8060-4df5-a62d-699d3a44caa9" containerName="maas-api" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173667 6221 cpu_manager.go:414] "containerMap: removing container" podUID="fd5ba5d9-23ee-42d4-97f0-3224353b90ec" containerName="payload-pre-processing" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173674 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd5ba5d9-23ee-42d4-97f0-3224353b90ec" containerName="payload-pre-processing" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173684 6221 cpu_manager.go:414] "containerMap: removing container" podUID="fd152ecb-4d3c-494c-83c1-182f018a4db1" containerName="istio-proxy" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173691 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd152ecb-4d3c-494c-83c1-182f018a4db1" containerName="istio-proxy" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173700 6221 cpu_manager.go:414] "containerMap: removing container" podUID="cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9" containerName="cleanup" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173708 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9" containerName="cleanup" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173716 6221 cpu_manager.go:414] "containerMap: removing container" podUID="fbab4591-a274-4c63-9004-8c05dd8676df" containerName="maas-api" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173723 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbab4591-a274-4c63-9004-8c05dd8676df" containerName="maas-api" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173738 6221 cpu_manager.go:414] "containerMap: removing container" podUID="decbaf90-f664-408c-a7df-b7f2667f399b" containerName="payload-pre-processing" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173746 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="decbaf90-f664-408c-a7df-b7f2667f399b" containerName="payload-pre-processing" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173757 6221 cpu_manager.go:414] "containerMap: removing container" podUID="1244d2b9-0346-460d-87bb-398ea191ee7b" containerName="main" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173765 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1244d2b9-0346-460d-87bb-398ea191ee7b" containerName="main" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173774 6221 cpu_manager.go:414] "containerMap: removing container" podUID="ebe12490-67c0-4d1d-ad75-1ade885d5650" containerName="cleanup" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173782 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebe12490-67c0-4d1d-ad75-1ade885d5650" containerName="cleanup" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173793 6221 cpu_manager.go:414] "containerMap: removing container" podUID="66320d4a-0c8f-483d-98e4-b17658b6b27a" containerName="cleanup" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173800 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66320d4a-0c8f-483d-98e4-b17658b6b27a" containerName="cleanup" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173810 6221 cpu_manager.go:414] "containerMap: removing container" podUID="dcd89198-b4a1-40cd-9850-3dda3af7bbab" containerName="istio-proxy" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173818 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcd89198-b4a1-40cd-9850-3dda3af7bbab" containerName="istio-proxy" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173831 6221 cpu_manager.go:414] "containerMap: removing container" podUID="fcbfae96-e276-4753-acb5-59ee55640aa8" containerName="payload-pre-processing" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173839 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcbfae96-e276-4753-acb5-59ee55640aa8" containerName="payload-pre-processing" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173849 6221 cpu_manager.go:414] "containerMap: removing container" podUID="a506dae9-1ced-409d-99db-413b66dd9d59" containerName="cleanup" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173859 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a506dae9-1ced-409d-99db-413b66dd9d59" containerName="cleanup" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173870 6221 cpu_manager.go:414] "containerMap: removing container" podUID="d0218485-8862-4117-902d-d5de5c8c963b" containerName="istio-proxy" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173878 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0218485-8862-4117-902d-d5de5c8c963b" containerName="istio-proxy" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173890 6221 cpu_manager.go:414] "containerMap: removing container" podUID="51dc5d9e-31aa-4246-9bd6-8953c55424f1" containerName="istio-proxy" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173897 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51dc5d9e-31aa-4246-9bd6-8953c55424f1" containerName="istio-proxy" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173907 6221 cpu_manager.go:414] "containerMap: removing container" podUID="d5b94b56-1785-45b7-ae04-3afb97ace8a3" containerName="maas-api" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173914 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5b94b56-1785-45b7-ae04-3afb97ace8a3" containerName="maas-api" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173926 6221 cpu_manager.go:414] "containerMap: removing container" podUID="cc95feca-80e8-41ac-8a29-be0b9c840d1d" containerName="payload-pre-processing" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173934 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc95feca-80e8-41ac-8a29-be0b9c840d1d" containerName="payload-pre-processing" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173944 6221 cpu_manager.go:414] "containerMap: removing container" podUID="c4ac5f29-bf87-42b2-82af-ee49cd16e879" containerName="cleanup" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173951 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4ac5f29-bf87-42b2-82af-ee49cd16e879" containerName="cleanup" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173966 6221 cpu_manager.go:414] "containerMap: removing container" podUID="0eec39a3-c8c5-47fd-b1c3-132da72e7b65" containerName="maas-api" Oct 30 02:21:51.173955 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173974 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eec39a3-c8c5-47fd-b1c3-132da72e7b65" containerName="maas-api" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173987 6221 cpu_manager.go:414] "containerMap: removing container" podUID="666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa" containerName="istio-proxy" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.173993 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa" containerName="istio-proxy" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174008 6221 cpu_manager.go:414] "containerMap: removing container" podUID="6c21d89b-1302-47e9-a329-ff6e4bd5e10e" containerName="payload-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174016 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c21d89b-1302-47e9-a329-ff6e4bd5e10e" containerName="payload-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174026 6221 cpu_manager.go:414] "containerMap: removing container" podUID="8e229797-d116-4670-86e9-e446396807ad" containerName="istio-proxy" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174033 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e229797-d116-4670-86e9-e446396807ad" containerName="istio-proxy" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174045 6221 cpu_manager.go:414] "containerMap: removing container" podUID="c4ed9612-c80c-4768-a30d-5d43f2868e89" containerName="payload-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174051 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4ed9612-c80c-4768-a30d-5d43f2868e89" containerName="payload-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174064 6221 cpu_manager.go:414] "containerMap: removing container" podUID="cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9" containerName="cleanup" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174072 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9" containerName="cleanup" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174084 6221 cpu_manager.go:414] "containerMap: removing container" podUID="125fc809-a755-460c-8605-363489a85b49" containerName="payload-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174091 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="125fc809-a755-460c-8605-363489a85b49" containerName="payload-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174101 6221 cpu_manager.go:414] "containerMap: removing container" podUID="fd152ecb-4d3c-494c-83c1-182f018a4db1" containerName="istio-proxy" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174108 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd152ecb-4d3c-494c-83c1-182f018a4db1" containerName="istio-proxy" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174117 6221 cpu_manager.go:414] "containerMap: removing container" podUID="28dc66b5-3755-4151-a2a3-4dbc793d0c28" containerName="payload-pre-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174125 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28dc66b5-3755-4151-a2a3-4dbc793d0c28" containerName="payload-pre-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174137 6221 cpu_manager.go:414] "containerMap: removing container" podUID="ab3bacd1-4d0c-4c6b-b672-582c8d729702" containerName="payload-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174144 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab3bacd1-4d0c-4c6b-b672-582c8d729702" containerName="payload-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174156 6221 cpu_manager.go:414] "containerMap: removing container" podUID="68d8d09d-2725-417b-8d9d-c0ea49ce29ac" containerName="cleanup" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174163 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68d8d09d-2725-417b-8d9d-c0ea49ce29ac" containerName="cleanup" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174174 6221 cpu_manager.go:414] "containerMap: removing container" podUID="08f6b864-02b6-4aa9-8616-777fb088d0a4" containerName="maas-api" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174183 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08f6b864-02b6-4aa9-8616-777fb088d0a4" containerName="maas-api" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174192 6221 cpu_manager.go:414] "containerMap: removing container" podUID="a506dae9-1ced-409d-99db-413b66dd9d59" containerName="cleanup" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174199 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a506dae9-1ced-409d-99db-413b66dd9d59" containerName="cleanup" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174208 6221 cpu_manager.go:414] "containerMap: removing container" podUID="1153eb7c-0a5d-4bae-8894-1d1519393611" containerName="payload-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174215 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1153eb7c-0a5d-4bae-8894-1d1519393611" containerName="payload-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174231 6221 cpu_manager.go:414] "containerMap: removing container" podUID="0c9898f9-d745-4345-97aa-7e82f95095c6" containerName="payload-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174238 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c9898f9-d745-4345-97aa-7e82f95095c6" containerName="payload-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174249 6221 cpu_manager.go:414] "containerMap: removing container" podUID="a36fc050-879a-4182-ac44-6d3ad2ff189e" containerName="payload-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174257 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a36fc050-879a-4182-ac44-6d3ad2ff189e" containerName="payload-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174267 6221 cpu_manager.go:414] "containerMap: removing container" podUID="8e229797-d116-4670-86e9-e446396807ad" containerName="istio-proxy" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174274 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e229797-d116-4670-86e9-e446396807ad" containerName="istio-proxy" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174286 6221 cpu_manager.go:414] "containerMap: removing container" podUID="a7f33e22-ce25-46bc-8add-b9aa4434e0e3" containerName="payload-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174295 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7f33e22-ce25-46bc-8add-b9aa4434e0e3" containerName="payload-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174306 6221 cpu_manager.go:414] "containerMap: removing container" podUID="d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2" containerName="maas-api" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174314 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2" containerName="maas-api" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174323 6221 cpu_manager.go:414] "containerMap: removing container" podUID="144c527d-0189-4c40-a64a-a207dfbdd710" containerName="cleanup" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174332 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="144c527d-0189-4c40-a64a-a207dfbdd710" containerName="cleanup" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174342 6221 cpu_manager.go:414] "containerMap: removing container" podUID="d90c1ed4-5a28-4ce1-ba6e-c521698c3667" containerName="cleanup" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174349 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d90c1ed4-5a28-4ce1-ba6e-c521698c3667" containerName="cleanup" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174360 6221 cpu_manager.go:414] "containerMap: removing container" podUID="30b6e290-ede2-4b9f-b65d-64d7e14355c5" containerName="payload-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174368 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30b6e290-ede2-4b9f-b65d-64d7e14355c5" containerName="payload-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174377 6221 cpu_manager.go:414] "containerMap: removing container" podUID="b124e8c4-1d49-4407-8841-99a368505a35" containerName="payload-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174384 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b124e8c4-1d49-4407-8841-99a368505a35" containerName="payload-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174393 6221 cpu_manager.go:414] "containerMap: removing container" podUID="de9bc101-0002-4af2-a1fa-ae9c6be2ff73" containerName="istio-proxy" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174400 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de9bc101-0002-4af2-a1fa-ae9c6be2ff73" containerName="istio-proxy" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174410 6221 cpu_manager.go:414] "containerMap: removing container" podUID="66320d4a-0c8f-483d-98e4-b17658b6b27a" containerName="cleanup" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174417 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66320d4a-0c8f-483d-98e4-b17658b6b27a" containerName="cleanup" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174429 6221 cpu_manager.go:414] "containerMap: removing container" podUID="a506dae9-1ced-409d-99db-413b66dd9d59" containerName="cleanup" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174437 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a506dae9-1ced-409d-99db-413b66dd9d59" containerName="cleanup" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174448 6221 cpu_manager.go:414] "containerMap: removing container" podUID="68d8d09d-2725-417b-8d9d-c0ea49ce29ac" containerName="cleanup" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174455 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68d8d09d-2725-417b-8d9d-c0ea49ce29ac" containerName="cleanup" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174468 6221 cpu_manager.go:414] "containerMap: removing container" podUID="a577ca5d-a4d6-4f5c-8f0e-422290e6ec61" containerName="payload-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174475 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a577ca5d-a4d6-4f5c-8f0e-422290e6ec61" containerName="payload-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174488 6221 cpu_manager.go:414] "containerMap: removing container" podUID="1c9548ee-90ef-4fe2-9412-4da31c20dfba" containerName="istio-proxy" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174495 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c9548ee-90ef-4fe2-9412-4da31c20dfba" containerName="istio-proxy" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174507 6221 cpu_manager.go:414] "containerMap: removing container" podUID="c4ac5f29-bf87-42b2-82af-ee49cd16e879" containerName="cleanup" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174515 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4ac5f29-bf87-42b2-82af-ee49cd16e879" containerName="cleanup" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174526 6221 cpu_manager.go:414] "containerMap: removing container" podUID="24f7ec30-91bf-4af8-a913-f1650e340227" containerName="maas-api" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174548 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24f7ec30-91bf-4af8-a913-f1650e340227" containerName="maas-api" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174560 6221 cpu_manager.go:414] "containerMap: removing container" podUID="7f16168b-61d0-4a08-94b7-7ea5581e4005" containerName="payload-pre-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174568 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f16168b-61d0-4a08-94b7-7ea5581e4005" containerName="payload-pre-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174579 6221 cpu_manager.go:414] "containerMap: removing container" podUID="61c55830-d642-4daa-8692-5f4f409b9a1a" containerName="istio-proxy" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174586 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61c55830-d642-4daa-8692-5f4f409b9a1a" containerName="istio-proxy" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174600 6221 cpu_manager.go:414] "containerMap: removing container" podUID="144c527d-0189-4c40-a64a-a207dfbdd710" containerName="cleanup" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174607 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="144c527d-0189-4c40-a64a-a207dfbdd710" containerName="cleanup" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174619 6221 cpu_manager.go:414] "containerMap: removing container" podUID="f5083acb-200e-4c21-8879-b7a1ebc8d337" containerName="payload-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174626 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5083acb-200e-4c21-8879-b7a1ebc8d337" containerName="payload-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174639 6221 cpu_manager.go:414] "containerMap: removing container" podUID="354ec282-b972-4fca-b4e3-1e33ef461f2f" containerName="main" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174647 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="354ec282-b972-4fca-b4e3-1e33ef461f2f" containerName="main" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174658 6221 cpu_manager.go:414] "containerMap: removing container" podUID="ea9583aa-8259-4c9b-932e-2d6559008d86" containerName="main" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174665 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea9583aa-8259-4c9b-932e-2d6559008d86" containerName="main" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174676 6221 cpu_manager.go:414] "containerMap: removing container" podUID="b2d4fdd6-a883-4f5c-90bd-6f36b799ffa0" containerName="payload-pre-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174683 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2d4fdd6-a883-4f5c-90bd-6f36b799ffa0" containerName="payload-pre-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174695 6221 cpu_manager.go:414] "containerMap: removing container" podUID="4cbfa236-b599-4625-bf0f-8d5d3e47398e" containerName="maas-api" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174702 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cbfa236-b599-4625-bf0f-8d5d3e47398e" containerName="maas-api" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174715 6221 cpu_manager.go:414] "containerMap: removing container" podUID="ebe12490-67c0-4d1d-ad75-1ade885d5650" containerName="cleanup" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174722 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebe12490-67c0-4d1d-ad75-1ade885d5650" containerName="cleanup" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174737 6221 cpu_manager.go:414] "containerMap: removing container" podUID="26457d5d-6b8f-4e85-8bc1-321115791efc" containerName="payload-pre-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174745 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26457d5d-6b8f-4e85-8bc1-321115791efc" containerName="payload-pre-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174756 6221 cpu_manager.go:414] "containerMap: removing container" podUID="a08e7d0c-aaf7-4960-bc25-23e2290c1bd6" containerName="payload-pre-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174784 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a08e7d0c-aaf7-4960-bc25-23e2290c1bd6" containerName="payload-pre-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174797 6221 cpu_manager.go:414] "containerMap: removing container" podUID="702e03b2-6e41-4a27-837a-ede7f1c2e2c2" containerName="payload-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174806 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="702e03b2-6e41-4a27-837a-ede7f1c2e2c2" containerName="payload-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174817 6221 cpu_manager.go:414] "containerMap: removing container" podUID="df8ed3ef-12e5-40a7-9322-4b2a5f921d60" containerName="payload-pre-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174825 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df8ed3ef-12e5-40a7-9322-4b2a5f921d60" containerName="payload-pre-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174835 6221 cpu_manager.go:414] "containerMap: removing container" podUID="144c527d-0189-4c40-a64a-a207dfbdd710" containerName="cleanup" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174842 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="144c527d-0189-4c40-a64a-a207dfbdd710" containerName="cleanup" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174947 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="d90c1ed4-5a28-4ce1-ba6e-c521698c3667" containerName="cleanup" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174960 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c9898f9-d745-4345-97aa-7e82f95095c6" containerName="payload-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174974 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="a36fc050-879a-4182-ac44-6d3ad2ff189e" containerName="payload-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174983 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e229797-d116-4670-86e9-e446396807ad" containerName="istio-proxy" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.174995 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7f33e22-ce25-46bc-8add-b9aa4434e0e3" containerName="payload-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175006 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8e484a7-e60d-4ea4-ba8e-ddd19081c5e2" containerName="maas-api" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175018 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="144c527d-0189-4c40-a64a-a207dfbdd710" containerName="cleanup" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175026 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="d90c1ed4-5a28-4ce1-ba6e-c521698c3667" containerName="cleanup" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175035 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="30b6e290-ede2-4b9f-b65d-64d7e14355c5" containerName="payload-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175045 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="b124e8c4-1d49-4407-8841-99a368505a35" containerName="payload-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175056 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="de9bc101-0002-4af2-a1fa-ae9c6be2ff73" containerName="istio-proxy" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175066 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="66320d4a-0c8f-483d-98e4-b17658b6b27a" containerName="cleanup" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175075 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="a506dae9-1ced-409d-99db-413b66dd9d59" containerName="cleanup" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175084 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="68d8d09d-2725-417b-8d9d-c0ea49ce29ac" containerName="cleanup" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175094 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="a577ca5d-a4d6-4f5c-8f0e-422290e6ec61" containerName="payload-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175104 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c9548ee-90ef-4fe2-9412-4da31c20dfba" containerName="istio-proxy" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175112 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4ac5f29-bf87-42b2-82af-ee49cd16e879" containerName="cleanup" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175121 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="24f7ec30-91bf-4af8-a913-f1650e340227" containerName="maas-api" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175131 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b685aa0-0c58-4911-8f31-3e920b8151a1" containerName="istio-proxy" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175140 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f16168b-61d0-4a08-94b7-7ea5581e4005" containerName="payload-pre-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175152 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="61c55830-d642-4daa-8692-5f4f409b9a1a" containerName="istio-proxy" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175162 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="144c527d-0189-4c40-a64a-a207dfbdd710" containerName="cleanup" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175171 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="68d8d09d-2725-417b-8d9d-c0ea49ce29ac" containerName="cleanup" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175180 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5083acb-200e-4c21-8879-b7a1ebc8d337" containerName="payload-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175189 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="354ec282-b972-4fca-b4e3-1e33ef461f2f" containerName="main" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175200 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2d4fdd6-a883-4f5c-90bd-6f36b799ffa0" containerName="payload-pre-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175210 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea9583aa-8259-4c9b-932e-2d6559008d86" containerName="main" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175219 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcd89198-b4a1-40cd-9850-3dda3af7bbab" containerName="istio-proxy" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175228 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5227b83-d834-4276-bd61-27e1251a0708" containerName="istio-proxy" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175236 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cbfa236-b599-4625-bf0f-8d5d3e47398e" containerName="maas-api" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175246 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9" containerName="cleanup" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175253 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebe12490-67c0-4d1d-ad75-1ade885d5650" containerName="cleanup" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175263 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="26457d5d-6b8f-4e85-8bc1-321115791efc" containerName="payload-pre-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175271 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="a08e7d0c-aaf7-4960-bc25-23e2290c1bd6" containerName="payload-pre-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175282 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0218485-8862-4117-902d-d5de5c8c963b" containerName="istio-proxy" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175290 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="702e03b2-6e41-4a27-837a-ede7f1c2e2c2" containerName="payload-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175298 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="df8ed3ef-12e5-40a7-9322-4b2a5f921d60" containerName="payload-pre-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175306 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebe12490-67c0-4d1d-ad75-1ade885d5650" containerName="cleanup" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175315 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="def2849e-c94c-4685-9992-e1121b744ac8" containerName="cleanup" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175325 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="9061008c-04d0-42d9-96ae-a8ebe16c54b7" containerName="payload-pre-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175336 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5227b83-d834-4276-bd61-27e1251a0708" containerName="istio-proxy" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175344 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="d90c1ed4-5a28-4ce1-ba6e-c521698c3667" containerName="cleanup" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175354 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="348f8c54-8ced-40e2-846d-f551e78b3092" containerName="maas-api" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175364 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="16bdf204-496b-4e72-b5cd-e7b5719043e5" containerName="payload-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175372 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="def2849e-c94c-4685-9992-e1121b744ac8" containerName="cleanup" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175382 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="def2849e-c94c-4685-9992-e1121b744ac8" containerName="cleanup" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175390 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa" containerName="istio-proxy" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175398 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9566007-9280-43dc-8c1d-0439ccc4f679" containerName="payload-pre-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175407 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="27948676-86ab-49f1-a4f7-3083a591738c" containerName="payload-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175417 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff0dc863-4ea6-454f-9f61-5bfd9a9db849" containerName="istio-proxy" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175426 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="c26c4344-4ccb-44d9-ad1d-cfd8720bc56d" containerName="maas-api" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175433 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="018c3fe2-3e67-45ed-a3d7-d70f3fe5b956" containerName="main" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175444 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="7562db53-8060-4df5-a62d-699d3a44caa9" containerName="maas-api" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175453 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd5ba5d9-23ee-42d4-97f0-3224353b90ec" containerName="payload-pre-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175461 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd152ecb-4d3c-494c-83c1-182f018a4db1" containerName="istio-proxy" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175469 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9" containerName="cleanup" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175478 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbab4591-a274-4c63-9004-8c05dd8676df" containerName="maas-api" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175487 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="decbaf90-f664-408c-a7df-b7f2667f399b" containerName="payload-pre-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175495 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="1244d2b9-0346-460d-87bb-398ea191ee7b" containerName="main" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175504 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebe12490-67c0-4d1d-ad75-1ade885d5650" containerName="cleanup" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175513 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcd89198-b4a1-40cd-9850-3dda3af7bbab" containerName="istio-proxy" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175523 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcbfae96-e276-4753-acb5-59ee55640aa8" containerName="payload-pre-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175545 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0218485-8862-4117-902d-d5de5c8c963b" containerName="istio-proxy" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175555 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="51dc5d9e-31aa-4246-9bd6-8953c55424f1" containerName="istio-proxy" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175562 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5b94b56-1785-45b7-ae04-3afb97ace8a3" containerName="maas-api" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175571 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc95feca-80e8-41ac-8a29-be0b9c840d1d" containerName="payload-pre-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175579 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4ac5f29-bf87-42b2-82af-ee49cd16e879" containerName="cleanup" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175588 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eec39a3-c8c5-47fd-b1c3-132da72e7b65" containerName="maas-api" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175596 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="666e3fe7-fc6e-4aaa-9f63-f6223c6d07aa" containerName="istio-proxy" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175610 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c21d89b-1302-47e9-a329-ff6e4bd5e10e" containerName="payload-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175619 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4ed9612-c80c-4768-a30d-5d43f2868e89" containerName="payload-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175628 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9" containerName="cleanup" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175638 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="125fc809-a755-460c-8605-363489a85b49" containerName="payload-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175647 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd152ecb-4d3c-494c-83c1-182f018a4db1" containerName="istio-proxy" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175654 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="28dc66b5-3755-4151-a2a3-4dbc793d0c28" containerName="payload-pre-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175664 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab3bacd1-4d0c-4c6b-b672-582c8d729702" containerName="payload-processing" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175673 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="66320d4a-0c8f-483d-98e4-b17658b6b27a" containerName="cleanup" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175682 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="08f6b864-02b6-4aa9-8616-777fb088d0a4" containerName="maas-api" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175691 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="a506dae9-1ced-409d-99db-413b66dd9d59" containerName="cleanup" Oct 30 02:21:51.176893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.175700 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="1153eb7c-0a5d-4bae-8894-1d1519393611" containerName="payload-processing" Oct 30 02:21:51.183474 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.180873 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07258647-dbt65" Oct 30 02:21:51.196430 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.196401 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07258647-dbt65"] Oct 30 02:21:51.280641 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.280610 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kkcjq\"" Oct 30 02:21:51.280641 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.280610 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 02:21:51.315489 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.315454 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2dc133d-f78e-4106-9957-422c963aa371-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07258647-dbt65\" (UID: \"f2dc133d-f78e-4106-9957-422c963aa371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07258647-dbt65" Oct 30 02:21:51.315675 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.315502 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29v8p\" (UniqueName: \"kubernetes.io/projected/f2dc133d-f78e-4106-9957-422c963aa371-kube-api-access-29v8p\") pod \"maas-api-revoke-sub-07258647-dbt65\" (UID: \"f2dc133d-f78e-4106-9957-422c963aa371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07258647-dbt65" Oct 30 02:21:51.315675 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.315645 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2dc133d-f78e-4106-9957-422c963aa371-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07258647-dbt65\" (UID: \"f2dc133d-f78e-4106-9957-422c963aa371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07258647-dbt65" Oct 30 02:21:51.416482 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.416441 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2dc133d-f78e-4106-9957-422c963aa371-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07258647-dbt65\" (UID: \"f2dc133d-f78e-4106-9957-422c963aa371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07258647-dbt65" Oct 30 02:21:51.416713 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.416568 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2dc133d-f78e-4106-9957-422c963aa371-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07258647-dbt65\" (UID: \"f2dc133d-f78e-4106-9957-422c963aa371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07258647-dbt65" Oct 30 02:21:51.416713 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.416605 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29v8p\" (UniqueName: \"kubernetes.io/projected/f2dc133d-f78e-4106-9957-422c963aa371-kube-api-access-29v8p\") pod \"maas-api-revoke-sub-07258647-dbt65\" (UID: \"f2dc133d-f78e-4106-9957-422c963aa371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07258647-dbt65" Oct 30 02:21:51.417270 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.417244 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2dc133d-f78e-4106-9957-422c963aa371-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07258647-dbt65\" (UID: \"f2dc133d-f78e-4106-9957-422c963aa371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07258647-dbt65" Oct 30 02:21:51.418894 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.418870 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2dc133d-f78e-4106-9957-422c963aa371-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07258647-dbt65\" (UID: \"f2dc133d-f78e-4106-9957-422c963aa371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07258647-dbt65" Oct 30 02:21:51.436457 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.436397 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29v8p\" (UniqueName: \"kubernetes.io/projected/f2dc133d-f78e-4106-9957-422c963aa371-kube-api-access-29v8p\") pod \"maas-api-revoke-sub-07258647-dbt65\" (UID: \"f2dc133d-f78e-4106-9957-422c963aa371\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07258647-dbt65" Oct 30 02:21:51.493369 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:21:51.493327 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 02:21:51.494580 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.494561 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07258647-dbt65" Oct 30 02:21:51.788217 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:51.788184 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07258647-dbt65"] Oct 30 02:21:51.791899 ip-10-0-135-65 kubenswrapper[6221]: W1030 02:21:51.791870 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2dc133d_f78e_4106_9957_422c963aa371.slice/crio-6fd69c3dd95c5143c1257715542c6a632e8f63da04c6910245a6e69041e70eec WatchSource:0}: Error finding container 6fd69c3dd95c5143c1257715542c6a632e8f63da04c6910245a6e69041e70eec: Status 404 returned error can't find the container with id 6fd69c3dd95c5143c1257715542c6a632e8f63da04c6910245a6e69041e70eec Oct 30 02:21:52.783613 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:52.783578 6221 generic.go:356] "Generic (PLEG): container finished" podID="f2dc133d-f78e-4106-9957-422c963aa371" containerID="5e7abccf5b4f3ce21cb3c6b459b4a796d036c9ded55eb872b68dfce27d64b1dd" exitCode=0 Oct 30 02:21:52.784061 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:52.783625 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07258647-dbt65" event={"ID":"f2dc133d-f78e-4106-9957-422c963aa371","Type":"ContainerDied","Data":"5e7abccf5b4f3ce21cb3c6b459b4a796d036c9ded55eb872b68dfce27d64b1dd"} Oct 30 02:21:52.784061 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:52.783664 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07258647-dbt65" event={"ID":"f2dc133d-f78e-4106-9957-422c963aa371","Type":"ContainerStarted","Data":"6fd69c3dd95c5143c1257715542c6a632e8f63da04c6910245a6e69041e70eec"} Oct 30 02:21:53.625402 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:53.625376 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-861cd2-6477f778c6-8cbg5" Oct 30 02:21:53.741329 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:53.741232 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4160a65-33dc-4c24-8a6d-f24c37cb3ebe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4160a65-33dc-4c24-8a6d-f24c37cb3ebe-plugins-config-volume\") pod \"e4160a65-33dc-4c24-8a6d-f24c37cb3ebe\" (UID: \"e4160a65-33dc-4c24-8a6d-f24c37cb3ebe\") " Oct 30 02:21:53.741329 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:53.741277 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4160a65-33dc-4c24-8a6d-f24c37cb3ebe-kube-api-access-hxrkx\" (UniqueName: \"kubernetes.io/projected/e4160a65-33dc-4c24-8a6d-f24c37cb3ebe-kube-api-access-hxrkx\") pod \"e4160a65-33dc-4c24-8a6d-f24c37cb3ebe\" (UID: \"e4160a65-33dc-4c24-8a6d-f24c37cb3ebe\") " Oct 30 02:21:53.741714 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:53.741691 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4160a65-33dc-4c24-8a6d-f24c37cb3ebe-plugins-config-volume" pod "e4160a65-33dc-4c24-8a6d-f24c37cb3ebe" (UID: "e4160a65-33dc-4c24-8a6d-f24c37cb3ebe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:21:53.743561 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:53.743515 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4160a65-33dc-4c24-8a6d-f24c37cb3ebe-kube-api-access-hxrkx" pod "e4160a65-33dc-4c24-8a6d-f24c37cb3ebe" (UID: "e4160a65-33dc-4c24-8a6d-f24c37cb3ebe"). InnerVolumeSpecName "kube-api-access-hxrkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:21:53.789837 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:53.789802 6221 generic.go:356] "Generic (PLEG): container finished" podID="e4160a65-33dc-4c24-8a6d-f24c37cb3ebe" containerID="78f3e7ed3ccaf97a4e8ee4df3cb62753455c143dd03264f121ea112d0763ce42" exitCode=0 Oct 30 02:21:53.790296 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:53.789850 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-861cd2-6477f778c6-8cbg5" Oct 30 02:21:53.790296 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:53.789890 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-861cd2-6477f778c6-8cbg5" event={"ID":"e4160a65-33dc-4c24-8a6d-f24c37cb3ebe","Type":"ContainerDied","Data":"78f3e7ed3ccaf97a4e8ee4df3cb62753455c143dd03264f121ea112d0763ce42"} Oct 30 02:21:53.790296 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:53.789942 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-861cd2-6477f778c6-8cbg5" event={"ID":"e4160a65-33dc-4c24-8a6d-f24c37cb3ebe","Type":"ContainerDied","Data":"d579666b42848b87f5c6456b9ec2d38265c1e6a7886d3ef59113763e62ab80da"} Oct 30 02:21:53.790296 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:53.789973 6221 scope.go:122] "RemoveContainer" containerID="78f3e7ed3ccaf97a4e8ee4df3cb62753455c143dd03264f121ea112d0763ce42" Oct 30 02:21:53.798993 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:53.798970 6221 scope.go:122] "RemoveContainer" containerID="78f3e7ed3ccaf97a4e8ee4df3cb62753455c143dd03264f121ea112d0763ce42" Oct 30 02:21:53.799264 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:21:53.799243 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78f3e7ed3ccaf97a4e8ee4df3cb62753455c143dd03264f121ea112d0763ce42\": container with ID starting with 78f3e7ed3ccaf97a4e8ee4df3cb62753455c143dd03264f121ea112d0763ce42 not found: ID does not exist" containerID="78f3e7ed3ccaf97a4e8ee4df3cb62753455c143dd03264f121ea112d0763ce42" Oct 30 02:21:53.799344 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:53.799277 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78f3e7ed3ccaf97a4e8ee4df3cb62753455c143dd03264f121ea112d0763ce42"} err="failed to get container status \"78f3e7ed3ccaf97a4e8ee4df3cb62753455c143dd03264f121ea112d0763ce42\": rpc error: code = NotFound desc = could not find container \"78f3e7ed3ccaf97a4e8ee4df3cb62753455c143dd03264f121ea112d0763ce42\": container with ID starting with 78f3e7ed3ccaf97a4e8ee4df3cb62753455c143dd03264f121ea112d0763ce42 not found: ID does not exist" Oct 30 02:21:53.842693 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:53.842662 6221 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4160a65-33dc-4c24-8a6d-f24c37cb3ebe-plugins-config-volume\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:21:53.842693 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:53.842690 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxrkx\" (UniqueName: \"kubernetes.io/projected/e4160a65-33dc-4c24-8a6d-f24c37cb3ebe-kube-api-access-hxrkx\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:21:53.845010 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:53.844981 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-861cd2-6477f778c6-8cbg5"] Oct 30 02:21:53.854201 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:53.854164 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-861cd2-6477f778c6-8cbg5"] Oct 30 02:21:53.919194 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:53.919168 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07258647-dbt65" Oct 30 02:21:54.044201 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:54.044172 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2dc133d-f78e-4106-9957-422c963aa371-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2dc133d-f78e-4106-9957-422c963aa371-maas-api-openshift-ca\") pod \"f2dc133d-f78e-4106-9957-422c963aa371\" (UID: \"f2dc133d-f78e-4106-9957-422c963aa371\") " Oct 30 02:21:54.044419 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:54.044232 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2dc133d-f78e-4106-9957-422c963aa371-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2dc133d-f78e-4106-9957-422c963aa371-maas-api-xks-ca\") pod \"f2dc133d-f78e-4106-9957-422c963aa371\" (UID: \"f2dc133d-f78e-4106-9957-422c963aa371\") " Oct 30 02:21:54.044419 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:54.044305 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2dc133d-f78e-4106-9957-422c963aa371-kube-api-access-29v8p\" (UniqueName: \"kubernetes.io/projected/f2dc133d-f78e-4106-9957-422c963aa371-kube-api-access-29v8p\") pod \"f2dc133d-f78e-4106-9957-422c963aa371\" (UID: \"f2dc133d-f78e-4106-9957-422c963aa371\") " Oct 30 02:21:54.044661 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:54.044635 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2dc133d-f78e-4106-9957-422c963aa371-maas-api-openshift-ca" pod "f2dc133d-f78e-4106-9957-422c963aa371" (UID: "f2dc133d-f78e-4106-9957-422c963aa371"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:21:54.044758 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:54.044732 6221 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2dc133d-f78e-4106-9957-422c963aa371-maas-api-openshift-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:21:54.046383 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:54.046358 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2dc133d-f78e-4106-9957-422c963aa371-maas-api-xks-ca" pod "f2dc133d-f78e-4106-9957-422c963aa371" (UID: "f2dc133d-f78e-4106-9957-422c963aa371"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:21:54.046479 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:54.046405 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2dc133d-f78e-4106-9957-422c963aa371-kube-api-access-29v8p" pod "f2dc133d-f78e-4106-9957-422c963aa371" (UID: "f2dc133d-f78e-4106-9957-422c963aa371"). InnerVolumeSpecName "kube-api-access-29v8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:21:54.145614 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:54.145580 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29v8p\" (UniqueName: \"kubernetes.io/projected/f2dc133d-f78e-4106-9957-422c963aa371-kube-api-access-29v8p\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:21:54.145614 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:54.145609 6221 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2dc133d-f78e-4106-9957-422c963aa371-maas-api-xks-ca\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:21:54.546007 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:54.545977 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4160a65-33dc-4c24-8a6d-f24c37cb3ebe" path="/var/lib/kubelet/pods/e4160a65-33dc-4c24-8a6d-f24c37cb3ebe/volumes" Oct 30 02:21:54.797268 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:54.797192 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07258647-dbt65" Oct 30 02:21:54.797268 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:54.797208 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07258647-dbt65" event={"ID":"f2dc133d-f78e-4106-9957-422c963aa371","Type":"ContainerDied","Data":"6fd69c3dd95c5143c1257715542c6a632e8f63da04c6910245a6e69041e70eec"} Oct 30 02:21:54.797268 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:54.797248 6221 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fd69c3dd95c5143c1257715542c6a632e8f63da04c6910245a6e69041e70eec" Oct 30 02:21:57.640319 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:57.640262 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" podUID="9df52a01-0c8d-44ba-9fd7-9cb2fa41947a" containerName="main" containerID="cri-o://bc8cfd1152fae94f9ecd92156cce1aec6f4ec04f8b2f9e8b155575e07647fb03" gracePeriod=45 Oct 30 02:21:57.645160 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:57.645073 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" podUID="3b3c0e08-1371-4184-816a-d8b501526190" containerName="main" containerID="cri-o://ab134ba62aec4bfcd7016f63343e4f5e2608ea01be89a49e2b74c24aeea0baaa" gracePeriod=45 Oct 30 02:21:57.687509 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:57.687471 6221 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" podUID="9df52a01-0c8d-44ba-9fd7-9cb2fa41947a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": dial tcp 10.134.0.68:8000: connect: connection refused" Oct 30 02:21:57.816946 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:57.816764 6221 generic.go:356] "Generic (PLEG): container finished" podID="9df52a01-0c8d-44ba-9fd7-9cb2fa41947a" containerID="bc8cfd1152fae94f9ecd92156cce1aec6f4ec04f8b2f9e8b155575e07647fb03" exitCode=0 Oct 30 02:21:57.816946 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:57.816875 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" event={"ID":"9df52a01-0c8d-44ba-9fd7-9cb2fa41947a","Type":"ContainerDied","Data":"bc8cfd1152fae94f9ecd92156cce1aec6f4ec04f8b2f9e8b155575e07647fb03"} Oct 30 02:21:57.819494 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:57.819461 6221 generic.go:356] "Generic (PLEG): container finished" podID="3b3c0e08-1371-4184-816a-d8b501526190" containerID="ab134ba62aec4bfcd7016f63343e4f5e2608ea01be89a49e2b74c24aeea0baaa" exitCode=0 Oct 30 02:21:57.819658 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:57.819517 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" event={"ID":"3b3c0e08-1371-4184-816a-d8b501526190","Type":"ContainerDied","Data":"ab134ba62aec4bfcd7016f63343e4f5e2608ea01be89a49e2b74c24aeea0baaa"} Oct 30 02:21:57.894658 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:57.894591 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" Oct 30 02:21:57.915198 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:57.915174 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" Oct 30 02:21:57.976480 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:57.976446 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b3c0e08-1371-4184-816a-d8b501526190-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b3c0e08-1371-4184-816a-d8b501526190-model-cache\") pod \"3b3c0e08-1371-4184-816a-d8b501526190\" (UID: \"3b3c0e08-1371-4184-816a-d8b501526190\") " Oct 30 02:21:57.976664 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:57.976524 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9df52a01-0c8d-44ba-9fd7-9cb2fa41947a-kube-api-access-dr87s\" (UniqueName: \"kubernetes.io/projected/9df52a01-0c8d-44ba-9fd7-9cb2fa41947a-kube-api-access-dr87s\") pod \"9df52a01-0c8d-44ba-9fd7-9cb2fa41947a\" (UID: \"9df52a01-0c8d-44ba-9fd7-9cb2fa41947a\") " Oct 30 02:21:57.976664 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:57.976573 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b3c0e08-1371-4184-816a-d8b501526190-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b3c0e08-1371-4184-816a-d8b501526190-dshm\") pod \"3b3c0e08-1371-4184-816a-d8b501526190\" (UID: \"3b3c0e08-1371-4184-816a-d8b501526190\") " Oct 30 02:21:57.976664 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:57.976615 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b3c0e08-1371-4184-816a-d8b501526190-kube-api-access-l4s49\" (UniqueName: \"kubernetes.io/projected/3b3c0e08-1371-4184-816a-d8b501526190-kube-api-access-l4s49\") pod \"3b3c0e08-1371-4184-816a-d8b501526190\" (UID: \"3b3c0e08-1371-4184-816a-d8b501526190\") " Oct 30 02:21:57.976664 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:57.976647 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9df52a01-0c8d-44ba-9fd7-9cb2fa41947a-tls-certs\" (UniqueName: \"kubernetes.io/secret/9df52a01-0c8d-44ba-9fd7-9cb2fa41947a-tls-certs\") pod \"9df52a01-0c8d-44ba-9fd7-9cb2fa41947a\" (UID: \"9df52a01-0c8d-44ba-9fd7-9cb2fa41947a\") " Oct 30 02:21:57.976923 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:57.976681 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df52a01-0c8d-44ba-9fd7-9cb2fa41947a-home\" (UniqueName: \"kubernetes.io/empty-dir/9df52a01-0c8d-44ba-9fd7-9cb2fa41947a-home\") pod \"9df52a01-0c8d-44ba-9fd7-9cb2fa41947a\" (UID: \"9df52a01-0c8d-44ba-9fd7-9cb2fa41947a\") " Oct 30 02:21:57.976923 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:57.976731 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b3c0e08-1371-4184-816a-d8b501526190-home\" (UniqueName: \"kubernetes.io/empty-dir/3b3c0e08-1371-4184-816a-d8b501526190-home\") pod \"3b3c0e08-1371-4184-816a-d8b501526190\" (UID: \"3b3c0e08-1371-4184-816a-d8b501526190\") " Oct 30 02:21:57.976923 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:57.976766 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df52a01-0c8d-44ba-9fd7-9cb2fa41947a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9df52a01-0c8d-44ba-9fd7-9cb2fa41947a-tmp-dir\") pod \"9df52a01-0c8d-44ba-9fd7-9cb2fa41947a\" (UID: \"9df52a01-0c8d-44ba-9fd7-9cb2fa41947a\") " Oct 30 02:21:57.976923 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:57.976774 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b3c0e08-1371-4184-816a-d8b501526190-model-cache" pod "3b3c0e08-1371-4184-816a-d8b501526190" (UID: "3b3c0e08-1371-4184-816a-d8b501526190"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:21:57.976923 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:57.976814 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b3c0e08-1371-4184-816a-d8b501526190-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b3c0e08-1371-4184-816a-d8b501526190-tmp-dir\") pod \"3b3c0e08-1371-4184-816a-d8b501526190\" (UID: \"3b3c0e08-1371-4184-816a-d8b501526190\") " Oct 30 02:21:57.976923 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:57.976863 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b3c0e08-1371-4184-816a-d8b501526190-tls-certs\" (UniqueName: \"kubernetes.io/secret/3b3c0e08-1371-4184-816a-d8b501526190-tls-certs\") pod \"3b3c0e08-1371-4184-816a-d8b501526190\" (UID: \"3b3c0e08-1371-4184-816a-d8b501526190\") " Oct 30 02:21:57.976923 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:57.976897 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df52a01-0c8d-44ba-9fd7-9cb2fa41947a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9df52a01-0c8d-44ba-9fd7-9cb2fa41947a-model-cache\") pod \"9df52a01-0c8d-44ba-9fd7-9cb2fa41947a\" (UID: \"9df52a01-0c8d-44ba-9fd7-9cb2fa41947a\") " Oct 30 02:21:57.979042 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:57.976944 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9df52a01-0c8d-44ba-9fd7-9cb2fa41947a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9df52a01-0c8d-44ba-9fd7-9cb2fa41947a-dshm\") pod \"9df52a01-0c8d-44ba-9fd7-9cb2fa41947a\" (UID: \"9df52a01-0c8d-44ba-9fd7-9cb2fa41947a\") " Oct 30 02:21:57.979042 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:57.976989 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df52a01-0c8d-44ba-9fd7-9cb2fa41947a-home" pod "9df52a01-0c8d-44ba-9fd7-9cb2fa41947a" (UID: "9df52a01-0c8d-44ba-9fd7-9cb2fa41947a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:21:57.979042 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:57.977068 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df52a01-0c8d-44ba-9fd7-9cb2fa41947a-tmp-dir" pod "9df52a01-0c8d-44ba-9fd7-9cb2fa41947a" (UID: "9df52a01-0c8d-44ba-9fd7-9cb2fa41947a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:21:57.979042 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:57.977264 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b3c0e08-1371-4184-816a-d8b501526190-home" pod "3b3c0e08-1371-4184-816a-d8b501526190" (UID: "3b3c0e08-1371-4184-816a-d8b501526190"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:21:57.979042 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:57.977409 6221 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9df52a01-0c8d-44ba-9fd7-9cb2fa41947a-home\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:21:57.979042 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:57.977456 6221 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b3c0e08-1371-4184-816a-d8b501526190-home\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:21:57.979042 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:57.977471 6221 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9df52a01-0c8d-44ba-9fd7-9cb2fa41947a-tmp-dir\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:21:57.979042 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:57.977498 6221 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b3c0e08-1371-4184-816a-d8b501526190-model-cache\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:21:57.979042 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:57.977518 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b3c0e08-1371-4184-816a-d8b501526190-tmp-dir" pod "3b3c0e08-1371-4184-816a-d8b501526190" (UID: "3b3c0e08-1371-4184-816a-d8b501526190"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:21:57.979042 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:57.978522 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df52a01-0c8d-44ba-9fd7-9cb2fa41947a-model-cache" pod "9df52a01-0c8d-44ba-9fd7-9cb2fa41947a" (UID: "9df52a01-0c8d-44ba-9fd7-9cb2fa41947a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:21:57.981141 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:57.980784 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9df52a01-0c8d-44ba-9fd7-9cb2fa41947a-kube-api-access-dr87s" pod "9df52a01-0c8d-44ba-9fd7-9cb2fa41947a" (UID: "9df52a01-0c8d-44ba-9fd7-9cb2fa41947a"). InnerVolumeSpecName "kube-api-access-dr87s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:21:57.981141 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:57.981086 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b3c0e08-1371-4184-816a-d8b501526190-dshm" pod "3b3c0e08-1371-4184-816a-d8b501526190" (UID: "3b3c0e08-1371-4184-816a-d8b501526190"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:21:57.982216 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:57.982191 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9df52a01-0c8d-44ba-9fd7-9cb2fa41947a-tls-certs" pod "9df52a01-0c8d-44ba-9fd7-9cb2fa41947a" (UID: "9df52a01-0c8d-44ba-9fd7-9cb2fa41947a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:21:57.982713 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:57.982691 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b3c0e08-1371-4184-816a-d8b501526190-kube-api-access-l4s49" pod "3b3c0e08-1371-4184-816a-d8b501526190" (UID: "3b3c0e08-1371-4184-816a-d8b501526190"). InnerVolumeSpecName "kube-api-access-l4s49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:21:57.982868 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:57.982847 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b3c0e08-1371-4184-816a-d8b501526190-tls-certs" pod "3b3c0e08-1371-4184-816a-d8b501526190" (UID: "3b3c0e08-1371-4184-816a-d8b501526190"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:21:57.987908 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:57.987880 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9df52a01-0c8d-44ba-9fd7-9cb2fa41947a-dshm" pod "9df52a01-0c8d-44ba-9fd7-9cb2fa41947a" (UID: "9df52a01-0c8d-44ba-9fd7-9cb2fa41947a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:21:58.078926 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:58.078883 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr87s\" (UniqueName: \"kubernetes.io/projected/9df52a01-0c8d-44ba-9fd7-9cb2fa41947a-kube-api-access-dr87s\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:21:58.078926 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:58.078918 6221 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b3c0e08-1371-4184-816a-d8b501526190-dshm\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:21:58.078926 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:58.078931 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4s49\" (UniqueName: \"kubernetes.io/projected/3b3c0e08-1371-4184-816a-d8b501526190-kube-api-access-l4s49\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:21:58.079174 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:58.078943 6221 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9df52a01-0c8d-44ba-9fd7-9cb2fa41947a-tls-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:21:58.079174 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:58.078955 6221 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b3c0e08-1371-4184-816a-d8b501526190-tmp-dir\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:21:58.079174 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:58.078965 6221 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b3c0e08-1371-4184-816a-d8b501526190-tls-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:21:58.079174 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:58.078976 6221 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9df52a01-0c8d-44ba-9fd7-9cb2fa41947a-model-cache\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:21:58.079174 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:58.078986 6221 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9df52a01-0c8d-44ba-9fd7-9cb2fa41947a-dshm\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:21:58.799331 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:58.799290 6221 scope.go:122] "RemoveContainer" containerID="bc8cfd1152fae94f9ecd92156cce1aec6f4ec04f8b2f9e8b155575e07647fb03" Oct 30 02:21:58.807749 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:58.807723 6221 scope.go:122] "RemoveContainer" containerID="ab134ba62aec4bfcd7016f63343e4f5e2608ea01be89a49e2b74c24aeea0baaa" Oct 30 02:21:58.824976 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:58.824957 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" Oct 30 02:21:58.825079 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:58.824992 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq" event={"ID":"9df52a01-0c8d-44ba-9fd7-9cb2fa41947a","Type":"ContainerDied","Data":"9102fa466d8ae33cafe2cc83e8c7128dac2f5e83489999fffa264c714146112b"} Oct 30 02:21:58.825995 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:58.825968 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" Oct 30 02:21:58.825995 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:58.825974 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t" event={"ID":"3b3c0e08-1371-4184-816a-d8b501526190","Type":"ContainerDied","Data":"8a60f7b53f0197c80d0a7429af1f726e2d870220457e850c289578735cdb3f09"} Oct 30 02:21:58.876020 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:58.875994 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq"] Oct 30 02:21:58.888194 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:58.888169 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-cf82f8/distinct-2-w1-cf82f8-kserve-7dd49b849d-qvbvq"] Oct 30 02:21:58.922597 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:58.922567 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t"] Oct 30 02:21:58.926064 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:21:58.926040 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-cf82f8/distinct-w1-cf82f8-kserve-c789bf4f-p9s8t"] Oct 30 02:22:00.550449 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:00.550411 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b3c0e08-1371-4184-816a-d8b501526190" path="/var/lib/kubelet/pods/3b3c0e08-1371-4184-816a-d8b501526190/volumes" Oct 30 02:22:00.550888 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:00.550873 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9df52a01-0c8d-44ba-9fd7-9cb2fa41947a" path="/var/lib/kubelet/pods/9df52a01-0c8d-44ba-9fd7-9cb2fa41947a/volumes" Oct 30 02:22:04.044216 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:04.044182 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 02:22:13.731246 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:13.731209 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-67lvt"] Oct 30 02:22:13.897417 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:13.894061 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q"] Oct 30 02:22:13.897417 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:13.894347 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q" podUID="efc2969c-be45-4d14-af7d-9f8f60e11317" containerName="istio-proxy" containerID="cri-o://cc5984444ae87516d39d551f30693269103f569ec72696325de23b17f3957fed" gracePeriod=30 Oct 30 02:22:16.154526 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:16.154493 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-67lvt_53979da9-49dd-440b-b647-bcc57e7f738c/limitador/0.log" Oct 30 02:22:17.200068 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:17.200039 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7ddbcdf875-d8wmr_daa56966-b092-4927-bac7-11fa18b39d00/manager/0.log" Oct 30 02:22:19.149055 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:19.149028 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q" Oct 30 02:22:19.287277 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:19.287251 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efc2969c-be45-4d14-af7d-9f8f60e11317-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc2969c-be45-4d14-af7d-9f8f60e11317-workload-socket\") pod \"efc2969c-be45-4d14-af7d-9f8f60e11317\" (UID: \"efc2969c-be45-4d14-af7d-9f8f60e11317\") " Oct 30 02:22:19.287470 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:19.287301 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efc2969c-be45-4d14-af7d-9f8f60e11317-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc2969c-be45-4d14-af7d-9f8f60e11317-credential-socket\") pod \"efc2969c-be45-4d14-af7d-9f8f60e11317\" (UID: \"efc2969c-be45-4d14-af7d-9f8f60e11317\") " Oct 30 02:22:19.287470 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:19.287367 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efc2969c-be45-4d14-af7d-9f8f60e11317-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efc2969c-be45-4d14-af7d-9f8f60e11317-istiod-ca-cert\") pod \"efc2969c-be45-4d14-af7d-9f8f60e11317\" (UID: \"efc2969c-be45-4d14-af7d-9f8f60e11317\") " Oct 30 02:22:19.287470 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:19.287444 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efc2969c-be45-4d14-af7d-9f8f60e11317-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efc2969c-be45-4d14-af7d-9f8f60e11317-workload-certs\") pod \"efc2969c-be45-4d14-af7d-9f8f60e11317\" (UID: \"efc2969c-be45-4d14-af7d-9f8f60e11317\") " Oct 30 02:22:19.287694 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:19.287484 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/efc2969c-be45-4d14-af7d-9f8f60e11317-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efc2969c-be45-4d14-af7d-9f8f60e11317-istio-podinfo\") pod \"efc2969c-be45-4d14-af7d-9f8f60e11317\" (UID: \"efc2969c-be45-4d14-af7d-9f8f60e11317\") " Oct 30 02:22:19.287694 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:19.287510 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efc2969c-be45-4d14-af7d-9f8f60e11317-kube-api-access-kfhpn\" (UniqueName: \"kubernetes.io/projected/efc2969c-be45-4d14-af7d-9f8f60e11317-kube-api-access-kfhpn\") pod \"efc2969c-be45-4d14-af7d-9f8f60e11317\" (UID: \"efc2969c-be45-4d14-af7d-9f8f60e11317\") " Oct 30 02:22:19.287694 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:19.287567 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efc2969c-be45-4d14-af7d-9f8f60e11317-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efc2969c-be45-4d14-af7d-9f8f60e11317-istio-data\") pod \"efc2969c-be45-4d14-af7d-9f8f60e11317\" (UID: \"efc2969c-be45-4d14-af7d-9f8f60e11317\") " Oct 30 02:22:19.287694 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:19.287595 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efc2969c-be45-4d14-af7d-9f8f60e11317-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efc2969c-be45-4d14-af7d-9f8f60e11317-istio-envoy\") pod \"efc2969c-be45-4d14-af7d-9f8f60e11317\" (UID: \"efc2969c-be45-4d14-af7d-9f8f60e11317\") " Oct 30 02:22:19.287694 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:19.287608 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc2969c-be45-4d14-af7d-9f8f60e11317-workload-socket" pod "efc2969c-be45-4d14-af7d-9f8f60e11317" (UID: "efc2969c-be45-4d14-af7d-9f8f60e11317"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:22:19.287694 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:19.287631 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efc2969c-be45-4d14-af7d-9f8f60e11317-istio-token\" (UniqueName: \"kubernetes.io/projected/efc2969c-be45-4d14-af7d-9f8f60e11317-istio-token\") pod \"efc2969c-be45-4d14-af7d-9f8f60e11317\" (UID: \"efc2969c-be45-4d14-af7d-9f8f60e11317\") " Oct 30 02:22:19.287995 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:19.287732 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc2969c-be45-4d14-af7d-9f8f60e11317-credential-socket" pod "efc2969c-be45-4d14-af7d-9f8f60e11317" (UID: "efc2969c-be45-4d14-af7d-9f8f60e11317"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:22:19.287995 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:19.287784 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc2969c-be45-4d14-af7d-9f8f60e11317-workload-certs" pod "efc2969c-be45-4d14-af7d-9f8f60e11317" (UID: "efc2969c-be45-4d14-af7d-9f8f60e11317"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:22:19.287995 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:19.287949 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc2969c-be45-4d14-af7d-9f8f60e11317-istio-data" pod "efc2969c-be45-4d14-af7d-9f8f60e11317" (UID: "efc2969c-be45-4d14-af7d-9f8f60e11317"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:22:19.287995 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:19.287965 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efc2969c-be45-4d14-af7d-9f8f60e11317-istiod-ca-cert" pod "efc2969c-be45-4d14-af7d-9f8f60e11317" (UID: "efc2969c-be45-4d14-af7d-9f8f60e11317"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:22:19.288141 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:19.288108 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efc2969c-be45-4d14-af7d-9f8f60e11317-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:22:19.288141 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:19.288131 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efc2969c-be45-4d14-af7d-9f8f60e11317-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:22:19.288261 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:19.288144 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc2969c-be45-4d14-af7d-9f8f60e11317-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:22:19.288261 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:19.288156 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efc2969c-be45-4d14-af7d-9f8f60e11317-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:22:19.288261 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:19.288170 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efc2969c-be45-4d14-af7d-9f8f60e11317-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:22:19.289946 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:19.289916 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/efc2969c-be45-4d14-af7d-9f8f60e11317-istio-podinfo" pod "efc2969c-be45-4d14-af7d-9f8f60e11317" (UID: "efc2969c-be45-4d14-af7d-9f8f60e11317"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:22:19.290051 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:19.289923 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efc2969c-be45-4d14-af7d-9f8f60e11317-istio-envoy" pod "efc2969c-be45-4d14-af7d-9f8f60e11317" (UID: "efc2969c-be45-4d14-af7d-9f8f60e11317"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:22:19.290236 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:19.290209 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efc2969c-be45-4d14-af7d-9f8f60e11317-istio-token" pod "efc2969c-be45-4d14-af7d-9f8f60e11317" (UID: "efc2969c-be45-4d14-af7d-9f8f60e11317"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:22:19.290236 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:19.290225 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efc2969c-be45-4d14-af7d-9f8f60e11317-kube-api-access-kfhpn" pod "efc2969c-be45-4d14-af7d-9f8f60e11317" (UID: "efc2969c-be45-4d14-af7d-9f8f60e11317"). InnerVolumeSpecName "kube-api-access-kfhpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:22:19.388877 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:19.388835 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efc2969c-be45-4d14-af7d-9f8f60e11317-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:22:19.388877 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:19.388875 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfhpn\" (UniqueName: \"kubernetes.io/projected/efc2969c-be45-4d14-af7d-9f8f60e11317-kube-api-access-kfhpn\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:22:19.388877 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:19.388887 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efc2969c-be45-4d14-af7d-9f8f60e11317-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:22:19.389108 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:19.388896 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efc2969c-be45-4d14-af7d-9f8f60e11317-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:22:19.935007 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:19.934970 6221 generic.go:356] "Generic (PLEG): container finished" podID="efc2969c-be45-4d14-af7d-9f8f60e11317" containerID="cc5984444ae87516d39d551f30693269103f569ec72696325de23b17f3957fed" exitCode=0 Oct 30 02:22:19.935193 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:19.935038 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q" Oct 30 02:22:19.935193 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:19.935055 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q" event={"ID":"efc2969c-be45-4d14-af7d-9f8f60e11317","Type":"ContainerDied","Data":"cc5984444ae87516d39d551f30693269103f569ec72696325de23b17f3957fed"} Oct 30 02:22:19.935193 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:19.935112 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q" event={"ID":"efc2969c-be45-4d14-af7d-9f8f60e11317","Type":"ContainerDied","Data":"d6c5025c312a62d132b9f7473e08880f4ce075f6e56c1be28d06a75c69a94d6e"} Oct 30 02:22:19.935193 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:19.935133 6221 scope.go:122] "RemoveContainer" containerID="cc5984444ae87516d39d551f30693269103f569ec72696325de23b17f3957fed" Oct 30 02:22:19.944302 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:19.944287 6221 scope.go:122] "RemoveContainer" containerID="cc5984444ae87516d39d551f30693269103f569ec72696325de23b17f3957fed" Oct 30 02:22:19.944608 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:22:19.944587 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc5984444ae87516d39d551f30693269103f569ec72696325de23b17f3957fed\": container with ID starting with cc5984444ae87516d39d551f30693269103f569ec72696325de23b17f3957fed not found: ID does not exist" containerID="cc5984444ae87516d39d551f30693269103f569ec72696325de23b17f3957fed" Oct 30 02:22:19.944676 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:19.944618 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc5984444ae87516d39d551f30693269103f569ec72696325de23b17f3957fed"} err="failed to get container status \"cc5984444ae87516d39d551f30693269103f569ec72696325de23b17f3957fed\": rpc error: code = NotFound desc = could not find container \"cc5984444ae87516d39d551f30693269103f569ec72696325de23b17f3957fed\": container with ID starting with cc5984444ae87516d39d551f30693269103f569ec72696325de23b17f3957fed not found: ID does not exist" Oct 30 02:22:19.999423 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:19.999387 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q"] Oct 30 02:22:20.011990 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:20.011959 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-cf82f8-openshift-default-75ff594697-5982q"] Oct 30 02:22:20.545596 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:20.545567 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efc2969c-be45-4d14-af7d-9f8f60e11317" path="/var/lib/kubelet/pods/efc2969c-be45-4d14-af7d-9f8f60e11317/volumes" Oct 30 02:22:21.493353 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:22:21.493314 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 02:22:30.316873 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:30.316844 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7ddbcdf875-d8wmr_daa56966-b092-4927-bac7-11fa18b39d00/manager/0.log" Oct 30 02:22:32.804275 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:32.804237 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7b8b8d5c6c-dm9dk_3236f15b-ffde-4a60-ba93-fb48f9befbb0/maas-api/0.log" Oct 30 02:22:33.040960 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:33.040934 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-07258647-dbt65_f2dc133d-f78e-4106-9957-422c963aa371/revoke-keys/0.log" Oct 30 02:22:33.391159 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:33.391080 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wh5tn"] Oct 30 02:22:33.391311 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:33.391267 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wh5tn" podUID="3a9c11a6-3a7f-473b-92c5-80a93a69bc34" containerName="discovery" containerID="cri-o://a87f5d94c19e00774fe96fec8ad35c20a8dfefc0a489d3410f3a92ea8b232a43" gracePeriod=30 Oct 30 02:22:33.641685 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:33.641626 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wh5tn" Oct 30 02:22:33.732868 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:33.732834 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a9c11a6-3a7f-473b-92c5-80a93a69bc34-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3a9c11a6-3a7f-473b-92c5-80a93a69bc34-istio-csr-ca-configmap\") pod \"3a9c11a6-3a7f-473b-92c5-80a93a69bc34\" (UID: \"3a9c11a6-3a7f-473b-92c5-80a93a69bc34\") " Oct 30 02:22:33.732868 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:33.732870 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a9c11a6-3a7f-473b-92c5-80a93a69bc34-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a9c11a6-3a7f-473b-92c5-80a93a69bc34-local-certs\") pod \"3a9c11a6-3a7f-473b-92c5-80a93a69bc34\" (UID: \"3a9c11a6-3a7f-473b-92c5-80a93a69bc34\") " Oct 30 02:22:33.733098 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:33.732920 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a9c11a6-3a7f-473b-92c5-80a93a69bc34-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3a9c11a6-3a7f-473b-92c5-80a93a69bc34-istio-csr-dns-cert\") pod \"3a9c11a6-3a7f-473b-92c5-80a93a69bc34\" (UID: \"3a9c11a6-3a7f-473b-92c5-80a93a69bc34\") " Oct 30 02:22:33.733098 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:33.732944 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a9c11a6-3a7f-473b-92c5-80a93a69bc34-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3a9c11a6-3a7f-473b-92c5-80a93a69bc34-istio-kubeconfig\") pod \"3a9c11a6-3a7f-473b-92c5-80a93a69bc34\" (UID: \"3a9c11a6-3a7f-473b-92c5-80a93a69bc34\") " Oct 30 02:22:33.733098 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:33.732974 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a9c11a6-3a7f-473b-92c5-80a93a69bc34-kube-api-access-w2xxq\" (UniqueName: \"kubernetes.io/projected/3a9c11a6-3a7f-473b-92c5-80a93a69bc34-kube-api-access-w2xxq\") pod \"3a9c11a6-3a7f-473b-92c5-80a93a69bc34\" (UID: \"3a9c11a6-3a7f-473b-92c5-80a93a69bc34\") " Oct 30 02:22:33.733098 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:33.733019 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a9c11a6-3a7f-473b-92c5-80a93a69bc34-cacerts\" (UniqueName: \"kubernetes.io/secret/3a9c11a6-3a7f-473b-92c5-80a93a69bc34-cacerts\") pod \"3a9c11a6-3a7f-473b-92c5-80a93a69bc34\" (UID: \"3a9c11a6-3a7f-473b-92c5-80a93a69bc34\") " Oct 30 02:22:33.733098 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:33.733046 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a9c11a6-3a7f-473b-92c5-80a93a69bc34-istio-token\" (UniqueName: \"kubernetes.io/projected/3a9c11a6-3a7f-473b-92c5-80a93a69bc34-istio-token\") pod \"3a9c11a6-3a7f-473b-92c5-80a93a69bc34\" (UID: \"3a9c11a6-3a7f-473b-92c5-80a93a69bc34\") " Oct 30 02:22:33.733389 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:33.733274 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a9c11a6-3a7f-473b-92c5-80a93a69bc34-istio-csr-ca-configmap" pod "3a9c11a6-3a7f-473b-92c5-80a93a69bc34" (UID: "3a9c11a6-3a7f-473b-92c5-80a93a69bc34"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:22:33.733460 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:33.733383 6221 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3a9c11a6-3a7f-473b-92c5-80a93a69bc34-istio-csr-ca-configmap\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:22:33.735516 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:33.735483 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a9c11a6-3a7f-473b-92c5-80a93a69bc34-cacerts" pod "3a9c11a6-3a7f-473b-92c5-80a93a69bc34" (UID: "3a9c11a6-3a7f-473b-92c5-80a93a69bc34"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:22:33.735790 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:33.735747 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a9c11a6-3a7f-473b-92c5-80a93a69bc34-istio-kubeconfig" pod "3a9c11a6-3a7f-473b-92c5-80a93a69bc34" (UID: "3a9c11a6-3a7f-473b-92c5-80a93a69bc34"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:22:33.735790 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:33.735774 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a9c11a6-3a7f-473b-92c5-80a93a69bc34-istio-csr-dns-cert" pod "3a9c11a6-3a7f-473b-92c5-80a93a69bc34" (UID: "3a9c11a6-3a7f-473b-92c5-80a93a69bc34"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:22:33.735936 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:33.735779 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a9c11a6-3a7f-473b-92c5-80a93a69bc34-istio-token" pod "3a9c11a6-3a7f-473b-92c5-80a93a69bc34" (UID: "3a9c11a6-3a7f-473b-92c5-80a93a69bc34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:22:33.735992 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:33.735942 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a9c11a6-3a7f-473b-92c5-80a93a69bc34-kube-api-access-w2xxq" pod "3a9c11a6-3a7f-473b-92c5-80a93a69bc34" (UID: "3a9c11a6-3a7f-473b-92c5-80a93a69bc34"). InnerVolumeSpecName "kube-api-access-w2xxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:22:33.736359 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:33.736336 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a9c11a6-3a7f-473b-92c5-80a93a69bc34-local-certs" pod "3a9c11a6-3a7f-473b-92c5-80a93a69bc34" (UID: "3a9c11a6-3a7f-473b-92c5-80a93a69bc34"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:22:33.834086 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:33.834052 6221 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3a9c11a6-3a7f-473b-92c5-80a93a69bc34-cacerts\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:22:33.834086 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:33.834083 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a9c11a6-3a7f-473b-92c5-80a93a69bc34-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:22:33.834086 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:33.834093 6221 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a9c11a6-3a7f-473b-92c5-80a93a69bc34-local-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:22:33.834781 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:33.834101 6221 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3a9c11a6-3a7f-473b-92c5-80a93a69bc34-istio-csr-dns-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:22:33.834781 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:33.834110 6221 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3a9c11a6-3a7f-473b-92c5-80a93a69bc34-istio-kubeconfig\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:22:33.834781 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:33.834119 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2xxq\" (UniqueName: \"kubernetes.io/projected/3a9c11a6-3a7f-473b-92c5-80a93a69bc34-kube-api-access-w2xxq\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:22:33.998736 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:33.998644 6221 generic.go:356] "Generic (PLEG): container finished" podID="3a9c11a6-3a7f-473b-92c5-80a93a69bc34" containerID="a87f5d94c19e00774fe96fec8ad35c20a8dfefc0a489d3410f3a92ea8b232a43" exitCode=0 Oct 30 02:22:33.998736 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:33.998702 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wh5tn" Oct 30 02:22:33.998736 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:33.998712 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wh5tn" event={"ID":"3a9c11a6-3a7f-473b-92c5-80a93a69bc34","Type":"ContainerDied","Data":"a87f5d94c19e00774fe96fec8ad35c20a8dfefc0a489d3410f3a92ea8b232a43"} Oct 30 02:22:33.999046 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:33.998778 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wh5tn" event={"ID":"3a9c11a6-3a7f-473b-92c5-80a93a69bc34","Type":"ContainerDied","Data":"e66d1f21ea4e397df49fee10e47a3dcab98fda7fa9f9fb6e36cd4f79cbd6247e"} Oct 30 02:22:33.999046 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:33.998805 6221 scope.go:122] "RemoveContainer" containerID="a87f5d94c19e00774fe96fec8ad35c20a8dfefc0a489d3410f3a92ea8b232a43" Oct 30 02:22:34.016960 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:34.016940 6221 scope.go:122] "RemoveContainer" containerID="a87f5d94c19e00774fe96fec8ad35c20a8dfefc0a489d3410f3a92ea8b232a43" Oct 30 02:22:34.017238 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:22:34.017218 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a87f5d94c19e00774fe96fec8ad35c20a8dfefc0a489d3410f3a92ea8b232a43\": container with ID starting with a87f5d94c19e00774fe96fec8ad35c20a8dfefc0a489d3410f3a92ea8b232a43 not found: ID does not exist" containerID="a87f5d94c19e00774fe96fec8ad35c20a8dfefc0a489d3410f3a92ea8b232a43" Oct 30 02:22:34.017279 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:34.017248 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a87f5d94c19e00774fe96fec8ad35c20a8dfefc0a489d3410f3a92ea8b232a43"} err="failed to get container status \"a87f5d94c19e00774fe96fec8ad35c20a8dfefc0a489d3410f3a92ea8b232a43\": rpc error: code = NotFound desc = could not find container \"a87f5d94c19e00774fe96fec8ad35c20a8dfefc0a489d3410f3a92ea8b232a43\": container with ID starting with a87f5d94c19e00774fe96fec8ad35c20a8dfefc0a489d3410f3a92ea8b232a43 not found: ID does not exist" Oct 30 02:22:34.048735 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:34.048709 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wh5tn"] Oct 30 02:22:34.051799 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:34.051777 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wh5tn"] Oct 30 02:22:34.544958 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:34.544925 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a9c11a6-3a7f-473b-92c5-80a93a69bc34" path="/var/lib/kubelet/pods/3a9c11a6-3a7f-473b-92c5-80a93a69bc34/volumes" Oct 30 02:22:37.393885 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:37.393858 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-67lvt_53979da9-49dd-440b-b647-bcc57e7f738c/limitador/0.log" Oct 30 02:22:38.741449 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:38.741374 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs"] Oct 30 02:22:38.741893 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:38.741602 6221 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs" podUID="161c7dc2-54ae-4cb9-9105-3035c5f4875f" containerName="istio-proxy" containerID="cri-o://8fc682e71c34336824667b2961312a35a807973ca69017380a267ccf2f509abe" gracePeriod=30 Oct 30 02:22:38.953213 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:38.953183 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66f46d7b58-rcnz2_6852625d-7a8f-42fe-a064-b811802f5daa/kube-auth-proxy/0.log" Oct 30 02:22:39.147090 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:39.147055 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-8mnlq_0f60b8d7-d3d0-4ce0-bd5b-28bd7afaadf4/istio-proxy/0.log" Oct 30 02:22:39.353626 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:39.353598 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-bqtxs_161c7dc2-54ae-4cb9-9105-3035c5f4875f/istio-proxy/0.log" Oct 30 02:22:39.790896 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:39.790866 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5bc884dd87-k8czs_b7614c59-c623-460a-adae-dfe0601605dd/istio-proxy/0.log" Oct 30 02:22:40.398484 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:40.398459 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-728tk_2c0844ce-ae06-46a1-8819-decfc87c21e0/payload-pre-processing/0.log" Oct 30 02:22:40.594336 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:40.594312 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-wp6mf_fc46a03a-6581-4d32-8ed7-a4d5c53625b4/payload-processing/0.log" Oct 30 02:22:41.572056 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:41.572030 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-s2bnf_8c94e2ef-848a-4530-a644-51419cdb6fbe/main/0.log" Oct 30 02:22:41.772647 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:41.772621 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-pmwss_94880b9b-8725-4337-aee3-0ec765960248/main/0.log" Oct 30 02:22:42.148639 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:42.148607 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-22l7k_1f3def97-cd3e-4553-abc8-35cbd888b408/main/0.log" Oct 30 02:22:42.344939 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:42.344911 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8ztcpp_34f45b5e-e31f-4309-92d4-039b539e1578/main/0.log" Oct 30 02:22:42.901637 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:42.901600 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-kpnsk_f4ae494e-81ec-4039-ae07-7ed1bf345f0f/main/0.log" Oct 30 02:22:43.989027 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:43.989002 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs" Oct 30 02:22:44.019305 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:44.019240 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/161c7dc2-54ae-4cb9-9105-3035c5f4875f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/161c7dc2-54ae-4cb9-9105-3035c5f4875f-istiod-ca-cert\") pod \"161c7dc2-54ae-4cb9-9105-3035c5f4875f\" (UID: \"161c7dc2-54ae-4cb9-9105-3035c5f4875f\") " Oct 30 02:22:44.019428 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:44.019310 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/161c7dc2-54ae-4cb9-9105-3035c5f4875f-kube-api-access-5jptk\" (UniqueName: \"kubernetes.io/projected/161c7dc2-54ae-4cb9-9105-3035c5f4875f-kube-api-access-5jptk\") pod \"161c7dc2-54ae-4cb9-9105-3035c5f4875f\" (UID: \"161c7dc2-54ae-4cb9-9105-3035c5f4875f\") " Oct 30 02:22:44.019428 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:44.019338 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/161c7dc2-54ae-4cb9-9105-3035c5f4875f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/161c7dc2-54ae-4cb9-9105-3035c5f4875f-istio-envoy\") pod \"161c7dc2-54ae-4cb9-9105-3035c5f4875f\" (UID: \"161c7dc2-54ae-4cb9-9105-3035c5f4875f\") " Oct 30 02:22:44.019579 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:44.019430 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/161c7dc2-54ae-4cb9-9105-3035c5f4875f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/161c7dc2-54ae-4cb9-9105-3035c5f4875f-istio-podinfo\") pod \"161c7dc2-54ae-4cb9-9105-3035c5f4875f\" (UID: \"161c7dc2-54ae-4cb9-9105-3035c5f4875f\") " Oct 30 02:22:44.019579 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:44.019468 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/161c7dc2-54ae-4cb9-9105-3035c5f4875f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/161c7dc2-54ae-4cb9-9105-3035c5f4875f-workload-certs\") pod \"161c7dc2-54ae-4cb9-9105-3035c5f4875f\" (UID: \"161c7dc2-54ae-4cb9-9105-3035c5f4875f\") " Oct 30 02:22:44.019579 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:44.019502 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/161c7dc2-54ae-4cb9-9105-3035c5f4875f-istio-token\" (UniqueName: \"kubernetes.io/projected/161c7dc2-54ae-4cb9-9105-3035c5f4875f-istio-token\") pod \"161c7dc2-54ae-4cb9-9105-3035c5f4875f\" (UID: \"161c7dc2-54ae-4cb9-9105-3035c5f4875f\") " Oct 30 02:22:44.019579 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:44.019554 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/161c7dc2-54ae-4cb9-9105-3035c5f4875f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/161c7dc2-54ae-4cb9-9105-3035c5f4875f-istio-data\") pod \"161c7dc2-54ae-4cb9-9105-3035c5f4875f\" (UID: \"161c7dc2-54ae-4cb9-9105-3035c5f4875f\") " Oct 30 02:22:44.019811 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:44.019596 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/161c7dc2-54ae-4cb9-9105-3035c5f4875f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/161c7dc2-54ae-4cb9-9105-3035c5f4875f-credential-socket\") pod \"161c7dc2-54ae-4cb9-9105-3035c5f4875f\" (UID: \"161c7dc2-54ae-4cb9-9105-3035c5f4875f\") " Oct 30 02:22:44.019811 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:44.019633 6221 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/161c7dc2-54ae-4cb9-9105-3035c5f4875f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/161c7dc2-54ae-4cb9-9105-3035c5f4875f-workload-socket\") pod \"161c7dc2-54ae-4cb9-9105-3035c5f4875f\" (UID: \"161c7dc2-54ae-4cb9-9105-3035c5f4875f\") " Oct 30 02:22:44.019811 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:44.019774 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/161c7dc2-54ae-4cb9-9105-3035c5f4875f-workload-certs" pod "161c7dc2-54ae-4cb9-9105-3035c5f4875f" (UID: "161c7dc2-54ae-4cb9-9105-3035c5f4875f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:22:44.020033 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:44.019817 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/161c7dc2-54ae-4cb9-9105-3035c5f4875f-istiod-ca-cert" pod "161c7dc2-54ae-4cb9-9105-3035c5f4875f" (UID: "161c7dc2-54ae-4cb9-9105-3035c5f4875f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:22:44.020033 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:44.019963 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/161c7dc2-54ae-4cb9-9105-3035c5f4875f-istio-data" pod "161c7dc2-54ae-4cb9-9105-3035c5f4875f" (UID: "161c7dc2-54ae-4cb9-9105-3035c5f4875f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:22:44.020033 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:44.019979 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/161c7dc2-54ae-4cb9-9105-3035c5f4875f-credential-socket" pod "161c7dc2-54ae-4cb9-9105-3035c5f4875f" (UID: "161c7dc2-54ae-4cb9-9105-3035c5f4875f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:22:44.020033 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:44.019981 6221 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/161c7dc2-54ae-4cb9-9105-3035c5f4875f-workload-certs\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:22:44.020033 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:44.020012 6221 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/161c7dc2-54ae-4cb9-9105-3035c5f4875f-istiod-ca-cert\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:22:44.020257 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:44.020141 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/161c7dc2-54ae-4cb9-9105-3035c5f4875f-workload-socket" pod "161c7dc2-54ae-4cb9-9105-3035c5f4875f" (UID: "161c7dc2-54ae-4cb9-9105-3035c5f4875f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:22:44.021743 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:44.021715 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/161c7dc2-54ae-4cb9-9105-3035c5f4875f-istio-token" pod "161c7dc2-54ae-4cb9-9105-3035c5f4875f" (UID: "161c7dc2-54ae-4cb9-9105-3035c5f4875f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:22:44.021828 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:44.021722 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/161c7dc2-54ae-4cb9-9105-3035c5f4875f-kube-api-access-5jptk" pod "161c7dc2-54ae-4cb9-9105-3035c5f4875f" (UID: "161c7dc2-54ae-4cb9-9105-3035c5f4875f"). InnerVolumeSpecName "kube-api-access-5jptk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:22:44.021990 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:44.021975 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/161c7dc2-54ae-4cb9-9105-3035c5f4875f-istio-podinfo" pod "161c7dc2-54ae-4cb9-9105-3035c5f4875f" (UID: "161c7dc2-54ae-4cb9-9105-3035c5f4875f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 02:22:44.022044 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:44.021993 6221 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/161c7dc2-54ae-4cb9-9105-3035c5f4875f-istio-envoy" pod "161c7dc2-54ae-4cb9-9105-3035c5f4875f" (UID: "161c7dc2-54ae-4cb9-9105-3035c5f4875f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:22:44.045392 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:44.045368 6221 generic.go:356] "Generic (PLEG): container finished" podID="161c7dc2-54ae-4cb9-9105-3035c5f4875f" containerID="8fc682e71c34336824667b2961312a35a807973ca69017380a267ccf2f509abe" exitCode=0 Oct 30 02:22:44.045505 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:44.045420 6221 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs" Oct 30 02:22:44.045505 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:44.045436 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs" event={"ID":"161c7dc2-54ae-4cb9-9105-3035c5f4875f","Type":"ContainerDied","Data":"8fc682e71c34336824667b2961312a35a807973ca69017380a267ccf2f509abe"} Oct 30 02:22:44.045505 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:44.045487 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs" event={"ID":"161c7dc2-54ae-4cb9-9105-3035c5f4875f","Type":"ContainerDied","Data":"bfd00aaecb02b812222fdb3f094a814b7efc53908600280de596a3dc6709be91"} Oct 30 02:22:44.045666 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:44.045510 6221 scope.go:122] "RemoveContainer" containerID="8fc682e71c34336824667b2961312a35a807973ca69017380a267ccf2f509abe" Oct 30 02:22:44.055080 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:44.055063 6221 scope.go:122] "RemoveContainer" containerID="8fc682e71c34336824667b2961312a35a807973ca69017380a267ccf2f509abe" Oct 30 02:22:44.055440 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:22:44.055412 6221 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fc682e71c34336824667b2961312a35a807973ca69017380a267ccf2f509abe\": container with ID starting with 8fc682e71c34336824667b2961312a35a807973ca69017380a267ccf2f509abe not found: ID does not exist" containerID="8fc682e71c34336824667b2961312a35a807973ca69017380a267ccf2f509abe" Oct 30 02:22:44.055549 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:44.055450 6221 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fc682e71c34336824667b2961312a35a807973ca69017380a267ccf2f509abe"} err="failed to get container status \"8fc682e71c34336824667b2961312a35a807973ca69017380a267ccf2f509abe\": rpc error: code = NotFound desc = could not find container \"8fc682e71c34336824667b2961312a35a807973ca69017380a267ccf2f509abe\": container with ID starting with 8fc682e71c34336824667b2961312a35a807973ca69017380a267ccf2f509abe not found: ID does not exist" Oct 30 02:22:44.098304 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:44.098272 6221 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs"] Oct 30 02:22:44.104062 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:44.104040 6221 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5bc884dd87-bqtxs"] Oct 30 02:22:44.121245 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:44.121222 6221 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/161c7dc2-54ae-4cb9-9105-3035c5f4875f-credential-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:22:44.121245 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:44.121243 6221 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/161c7dc2-54ae-4cb9-9105-3035c5f4875f-workload-socket\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:22:44.121380 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:44.121254 6221 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jptk\" (UniqueName: \"kubernetes.io/projected/161c7dc2-54ae-4cb9-9105-3035c5f4875f-kube-api-access-5jptk\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:22:44.121380 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:44.121264 6221 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/161c7dc2-54ae-4cb9-9105-3035c5f4875f-istio-envoy\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:22:44.121380 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:44.121273 6221 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/161c7dc2-54ae-4cb9-9105-3035c5f4875f-istio-podinfo\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:22:44.121380 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:44.121281 6221 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/161c7dc2-54ae-4cb9-9105-3035c5f4875f-istio-token\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:22:44.121380 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:44.121288 6221 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/161c7dc2-54ae-4cb9-9105-3035c5f4875f-istio-data\") on node \"ip-10-0-135-65.ec2.internal\" DevicePath \"\"" Oct 30 02:22:44.546867 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:44.546835 6221 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="161c7dc2-54ae-4cb9-9105-3035c5f4875f" path="/var/lib/kubelet/pods/161c7dc2-54ae-4cb9-9105-3035c5f4875f/volumes" Oct 30 02:22:51.493110 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:22:51.493073 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 02:22:53.859934 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.859902 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8hrdq/must-gather-qgd6x"] Oct 30 02:22:53.860342 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860276 6221 cpu_manager.go:414] "containerMap: removing container" podUID="3b3c0e08-1371-4184-816a-d8b501526190" containerName="main" Oct 30 02:22:53.860342 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860288 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b3c0e08-1371-4184-816a-d8b501526190" containerName="main" Oct 30 02:22:53.860342 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860302 6221 cpu_manager.go:414] "containerMap: removing container" podUID="a5227b83-d834-4276-bd61-27e1251a0708" containerName="istio-proxy" Oct 30 02:22:53.860342 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860308 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5227b83-d834-4276-bd61-27e1251a0708" containerName="istio-proxy" Oct 30 02:22:53.860342 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860320 6221 cpu_manager.go:414] "containerMap: removing container" podUID="d90c1ed4-5a28-4ce1-ba6e-c521698c3667" containerName="cleanup" Oct 30 02:22:53.860342 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860326 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d90c1ed4-5a28-4ce1-ba6e-c521698c3667" containerName="cleanup" Oct 30 02:22:53.860342 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860333 6221 cpu_manager.go:414] "containerMap: removing container" podUID="6b685aa0-0c58-4911-8f31-3e920b8151a1" containerName="istio-proxy" Oct 30 02:22:53.860342 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860338 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b685aa0-0c58-4911-8f31-3e920b8151a1" containerName="istio-proxy" Oct 30 02:22:53.860342 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860345 6221 cpu_manager.go:414] "containerMap: removing container" podUID="def2849e-c94c-4685-9992-e1121b744ac8" containerName="cleanup" Oct 30 02:22:53.860703 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860350 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="def2849e-c94c-4685-9992-e1121b744ac8" containerName="cleanup" Oct 30 02:22:53.860703 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860362 6221 cpu_manager.go:414] "containerMap: removing container" podUID="9df52a01-0c8d-44ba-9fd7-9cb2fa41947a" containerName="main" Oct 30 02:22:53.860703 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860367 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9df52a01-0c8d-44ba-9fd7-9cb2fa41947a" containerName="main" Oct 30 02:22:53.860703 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860374 6221 cpu_manager.go:414] "containerMap: removing container" podUID="3a9c11a6-3a7f-473b-92c5-80a93a69bc34" containerName="discovery" Oct 30 02:22:53.860703 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860379 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a9c11a6-3a7f-473b-92c5-80a93a69bc34" containerName="discovery" Oct 30 02:22:53.860703 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860384 6221 cpu_manager.go:414] "containerMap: removing container" podUID="161c7dc2-54ae-4cb9-9105-3035c5f4875f" containerName="istio-proxy" Oct 30 02:22:53.860703 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860389 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="161c7dc2-54ae-4cb9-9105-3035c5f4875f" containerName="istio-proxy" Oct 30 02:22:53.860703 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860395 6221 cpu_manager.go:414] "containerMap: removing container" podUID="ebe12490-67c0-4d1d-ad75-1ade885d5650" containerName="cleanup" Oct 30 02:22:53.860703 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860400 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebe12490-67c0-4d1d-ad75-1ade885d5650" containerName="cleanup" Oct 30 02:22:53.860703 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860407 6221 cpu_manager.go:414] "containerMap: removing container" podUID="dcd89198-b4a1-40cd-9850-3dda3af7bbab" containerName="istio-proxy" Oct 30 02:22:53.860703 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860411 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcd89198-b4a1-40cd-9850-3dda3af7bbab" containerName="istio-proxy" Oct 30 02:22:53.860703 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860420 6221 cpu_manager.go:414] "containerMap: removing container" podUID="d0218485-8862-4117-902d-d5de5c8c963b" containerName="istio-proxy" Oct 30 02:22:53.860703 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860425 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0218485-8862-4117-902d-d5de5c8c963b" containerName="istio-proxy" Oct 30 02:22:53.860703 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860431 6221 cpu_manager.go:414] "containerMap: removing container" podUID="51dc5d9e-31aa-4246-9bd6-8953c55424f1" containerName="istio-proxy" Oct 30 02:22:53.860703 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860435 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51dc5d9e-31aa-4246-9bd6-8953c55424f1" containerName="istio-proxy" Oct 30 02:22:53.860703 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860446 6221 cpu_manager.go:414] "containerMap: removing container" podUID="cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9" containerName="cleanup" Oct 30 02:22:53.860703 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860451 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc0a37d6-fc06-482d-abda-0f1d0ffcc2f9" containerName="cleanup" Oct 30 02:22:53.860703 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860459 6221 cpu_manager.go:414] "containerMap: removing container" podUID="efc2969c-be45-4d14-af7d-9f8f60e11317" containerName="istio-proxy" Oct 30 02:22:53.860703 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860464 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efc2969c-be45-4d14-af7d-9f8f60e11317" containerName="istio-proxy" Oct 30 02:22:53.860703 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860470 6221 cpu_manager.go:414] "containerMap: removing container" podUID="f2dc133d-f78e-4106-9957-422c963aa371" containerName="revoke-keys" Oct 30 02:22:53.860703 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860474 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2dc133d-f78e-4106-9957-422c963aa371" containerName="revoke-keys" Oct 30 02:22:53.860703 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860485 6221 cpu_manager.go:414] "containerMap: removing container" podUID="66320d4a-0c8f-483d-98e4-b17658b6b27a" containerName="cleanup" Oct 30 02:22:53.860703 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860490 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66320d4a-0c8f-483d-98e4-b17658b6b27a" containerName="cleanup" Oct 30 02:22:53.860703 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860496 6221 cpu_manager.go:414] "containerMap: removing container" podUID="68d8d09d-2725-417b-8d9d-c0ea49ce29ac" containerName="cleanup" Oct 30 02:22:53.860703 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860501 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68d8d09d-2725-417b-8d9d-c0ea49ce29ac" containerName="cleanup" Oct 30 02:22:53.860703 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860511 6221 cpu_manager.go:414] "containerMap: removing container" podUID="e4160a65-33dc-4c24-8a6d-f24c37cb3ebe" containerName="payload-processing" Oct 30 02:22:53.860703 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860516 6221 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4160a65-33dc-4c24-8a6d-f24c37cb3ebe" containerName="payload-processing" Oct 30 02:22:53.860703 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860584 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="66320d4a-0c8f-483d-98e4-b17658b6b27a" containerName="cleanup" Oct 30 02:22:53.860703 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860595 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="a506dae9-1ced-409d-99db-413b66dd9d59" containerName="cleanup" Oct 30 02:22:53.860703 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860604 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4ac5f29-bf87-42b2-82af-ee49cd16e879" containerName="cleanup" Oct 30 02:22:53.860703 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860612 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4160a65-33dc-4c24-8a6d-f24c37cb3ebe" containerName="payload-processing" Oct 30 02:22:53.860703 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860617 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b685aa0-0c58-4911-8f31-3e920b8151a1" containerName="istio-proxy" Oct 30 02:22:53.860703 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860624 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="144c527d-0189-4c40-a64a-a207dfbdd710" containerName="cleanup" Oct 30 02:22:53.860703 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860630 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="68d8d09d-2725-417b-8d9d-c0ea49ce29ac" containerName="cleanup" Oct 30 02:22:53.860703 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860637 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b3c0e08-1371-4184-816a-d8b501526190" containerName="main" Oct 30 02:22:53.860703 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860644 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="9df52a01-0c8d-44ba-9fd7-9cb2fa41947a" containerName="main" Oct 30 02:22:53.860703 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860650 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a9c11a6-3a7f-473b-92c5-80a93a69bc34" containerName="discovery" Oct 30 02:22:53.860703 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860655 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="161c7dc2-54ae-4cb9-9105-3035c5f4875f" containerName="istio-proxy" Oct 30 02:22:53.860703 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860661 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="51dc5d9e-31aa-4246-9bd6-8953c55424f1" containerName="istio-proxy" Oct 30 02:22:53.860703 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860668 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="efc2969c-be45-4d14-af7d-9f8f60e11317" containerName="istio-proxy" Oct 30 02:22:53.860703 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860674 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2dc133d-f78e-4106-9957-422c963aa371" containerName="revoke-keys" Oct 30 02:22:53.860703 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.860679 6221 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e229797-d116-4670-86e9-e446396807ad" containerName="istio-proxy" Oct 30 02:22:53.865401 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.865384 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8hrdq/must-gather-qgd6x" Oct 30 02:22:53.895005 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.894978 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8hrdq\"/\"default-dockercfg-9gr6v\"" Oct 30 02:22:53.895239 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.895218 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8hrdq\"/\"kube-root-ca.crt\"" Oct 30 02:22:53.895355 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.895225 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8hrdq\"/\"openshift-service-ca.crt\"" Oct 30 02:22:53.896584 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.896561 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8hrdq/must-gather-qgd6x"] Oct 30 02:22:53.909872 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.909849 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7thts\" (UniqueName: \"kubernetes.io/projected/753aceb1-1d1f-476a-8cc2-8b6196a6f587-kube-api-access-7thts\") pod \"must-gather-qgd6x\" (UID: \"753aceb1-1d1f-476a-8cc2-8b6196a6f587\") " pod="openshift-must-gather-8hrdq/must-gather-qgd6x" Oct 30 02:22:53.909974 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.909890 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/753aceb1-1d1f-476a-8cc2-8b6196a6f587-must-gather-output\") pod \"must-gather-qgd6x\" (UID: \"753aceb1-1d1f-476a-8cc2-8b6196a6f587\") " pod="openshift-must-gather-8hrdq/must-gather-qgd6x" Oct 30 02:22:53.952739 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.952706 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9dvcx/must-gather-v7cch"] Oct 30 02:22:53.956415 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.956395 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-lphpb/must-gather-8ggng"] Oct 30 02:22:53.956570 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.956553 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9dvcx/must-gather-v7cch" Oct 30 02:22:53.959737 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.959721 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lphpb/must-gather-8ggng" Oct 30 02:22:53.975051 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.975025 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9dvcx/must-gather-v7cch"] Oct 30 02:22:53.977549 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.977509 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lphpb/must-gather-8ggng"] Oct 30 02:22:53.982800 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.982770 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lphpb\"/\"openshift-service-ca.crt\"" Oct 30 02:22:53.982915 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.982796 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9dvcx\"/\"openshift-service-ca.crt\"" Oct 30 02:22:53.982915 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.982861 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-lphpb\"/\"kube-root-ca.crt\"" Oct 30 02:22:53.983045 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.982916 6221 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9dvcx\"/\"kube-root-ca.crt\"" Oct 30 02:22:53.983045 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.982945 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-lphpb\"/\"default-dockercfg-hrx8f\"" Oct 30 02:22:53.983045 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:53.982959 6221 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9dvcx\"/\"default-dockercfg-v8hcx\"" Oct 30 02:22:54.010661 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:54.010634 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7thts\" (UniqueName: \"kubernetes.io/projected/753aceb1-1d1f-476a-8cc2-8b6196a6f587-kube-api-access-7thts\") pod \"must-gather-qgd6x\" (UID: \"753aceb1-1d1f-476a-8cc2-8b6196a6f587\") " pod="openshift-must-gather-8hrdq/must-gather-qgd6x" Oct 30 02:22:54.010816 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:54.010715 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/753aceb1-1d1f-476a-8cc2-8b6196a6f587-must-gather-output\") pod \"must-gather-qgd6x\" (UID: \"753aceb1-1d1f-476a-8cc2-8b6196a6f587\") " pod="openshift-must-gather-8hrdq/must-gather-qgd6x" Oct 30 02:22:54.010816 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:54.010751 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e7b98bc3-782c-40c3-864e-bf5efd1ce565-must-gather-output\") pod \"must-gather-8ggng\" (UID: \"e7b98bc3-782c-40c3-864e-bf5efd1ce565\") " pod="openshift-must-gather-lphpb/must-gather-8ggng" Oct 30 02:22:54.010816 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:54.010787 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8fa42939-6f1b-43db-b11d-222cdc11f6bd-must-gather-output\") pod \"must-gather-v7cch\" (UID: \"8fa42939-6f1b-43db-b11d-222cdc11f6bd\") " pod="openshift-must-gather-9dvcx/must-gather-v7cch" Oct 30 02:22:54.010930 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:54.010860 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t824\" (UniqueName: \"kubernetes.io/projected/e7b98bc3-782c-40c3-864e-bf5efd1ce565-kube-api-access-4t824\") pod \"must-gather-8ggng\" (UID: \"e7b98bc3-782c-40c3-864e-bf5efd1ce565\") " pod="openshift-must-gather-lphpb/must-gather-8ggng" Oct 30 02:22:54.010930 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:54.010893 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd45g\" (UniqueName: \"kubernetes.io/projected/8fa42939-6f1b-43db-b11d-222cdc11f6bd-kube-api-access-vd45g\") pod \"must-gather-v7cch\" (UID: \"8fa42939-6f1b-43db-b11d-222cdc11f6bd\") " pod="openshift-must-gather-9dvcx/must-gather-v7cch" Oct 30 02:22:54.011141 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:54.011123 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/753aceb1-1d1f-476a-8cc2-8b6196a6f587-must-gather-output\") pod \"must-gather-qgd6x\" (UID: \"753aceb1-1d1f-476a-8cc2-8b6196a6f587\") " pod="openshift-must-gather-8hrdq/must-gather-qgd6x" Oct 30 02:22:54.030661 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:54.030642 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7thts\" (UniqueName: \"kubernetes.io/projected/753aceb1-1d1f-476a-8cc2-8b6196a6f587-kube-api-access-7thts\") pod \"must-gather-qgd6x\" (UID: \"753aceb1-1d1f-476a-8cc2-8b6196a6f587\") " pod="openshift-must-gather-8hrdq/must-gather-qgd6x" Oct 30 02:22:54.112036 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:54.111958 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e7b98bc3-782c-40c3-864e-bf5efd1ce565-must-gather-output\") pod \"must-gather-8ggng\" (UID: \"e7b98bc3-782c-40c3-864e-bf5efd1ce565\") " pod="openshift-must-gather-lphpb/must-gather-8ggng" Oct 30 02:22:54.112036 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:54.111997 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8fa42939-6f1b-43db-b11d-222cdc11f6bd-must-gather-output\") pod \"must-gather-v7cch\" (UID: \"8fa42939-6f1b-43db-b11d-222cdc11f6bd\") " pod="openshift-must-gather-9dvcx/must-gather-v7cch" Oct 30 02:22:54.112036 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:54.112022 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t824\" (UniqueName: \"kubernetes.io/projected/e7b98bc3-782c-40c3-864e-bf5efd1ce565-kube-api-access-4t824\") pod \"must-gather-8ggng\" (UID: \"e7b98bc3-782c-40c3-864e-bf5efd1ce565\") " pod="openshift-must-gather-lphpb/must-gather-8ggng" Oct 30 02:22:54.112256 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:54.112044 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd45g\" (UniqueName: \"kubernetes.io/projected/8fa42939-6f1b-43db-b11d-222cdc11f6bd-kube-api-access-vd45g\") pod \"must-gather-v7cch\" (UID: \"8fa42939-6f1b-43db-b11d-222cdc11f6bd\") " pod="openshift-must-gather-9dvcx/must-gather-v7cch" Oct 30 02:22:54.112329 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:54.112309 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e7b98bc3-782c-40c3-864e-bf5efd1ce565-must-gather-output\") pod \"must-gather-8ggng\" (UID: \"e7b98bc3-782c-40c3-864e-bf5efd1ce565\") " pod="openshift-must-gather-lphpb/must-gather-8ggng" Oct 30 02:22:54.112369 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:54.112337 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8fa42939-6f1b-43db-b11d-222cdc11f6bd-must-gather-output\") pod \"must-gather-v7cch\" (UID: \"8fa42939-6f1b-43db-b11d-222cdc11f6bd\") " pod="openshift-must-gather-9dvcx/must-gather-v7cch" Oct 30 02:22:54.132652 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:54.132624 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t824\" (UniqueName: \"kubernetes.io/projected/e7b98bc3-782c-40c3-864e-bf5efd1ce565-kube-api-access-4t824\") pod \"must-gather-8ggng\" (UID: \"e7b98bc3-782c-40c3-864e-bf5efd1ce565\") " pod="openshift-must-gather-lphpb/must-gather-8ggng" Oct 30 02:22:54.132784 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:54.132657 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd45g\" (UniqueName: \"kubernetes.io/projected/8fa42939-6f1b-43db-b11d-222cdc11f6bd-kube-api-access-vd45g\") pod \"must-gather-v7cch\" (UID: \"8fa42939-6f1b-43db-b11d-222cdc11f6bd\") " pod="openshift-must-gather-9dvcx/must-gather-v7cch" Oct 30 02:22:54.179007 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:54.178971 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8hrdq/must-gather-qgd6x" Oct 30 02:22:54.271904 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:54.271875 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9dvcx/must-gather-v7cch" Oct 30 02:22:54.276293 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:54.276268 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-lphpb/must-gather-8ggng" Oct 30 02:22:54.476266 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:54.476193 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8hrdq/must-gather-qgd6x"] Oct 30 02:22:54.482782 ip-10-0-135-65 kubenswrapper[6221]: W1030 02:22:54.482740 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod753aceb1_1d1f_476a_8cc2_8b6196a6f587.slice/crio-fa2122afc2264d48e5ef04454ce09a4214c70da2716da51b28b7262d75bdc443 WatchSource:0}: Error finding container fa2122afc2264d48e5ef04454ce09a4214c70da2716da51b28b7262d75bdc443: Status 404 returned error can't find the container with id fa2122afc2264d48e5ef04454ce09a4214c70da2716da51b28b7262d75bdc443 Oct 30 02:22:54.484336 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:54.484317 6221 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 02:22:54.572849 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:54.572500 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-lphpb/must-gather-8ggng"] Oct 30 02:22:54.576570 ip-10-0-135-65 kubenswrapper[6221]: W1030 02:22:54.576543 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7b98bc3_782c_40c3_864e_bf5efd1ce565.slice/crio-9667035ee2759f9d1a63564b4f5f49b34df1ae4aa3124436e270268383197352 WatchSource:0}: Error finding container 9667035ee2759f9d1a63564b4f5f49b34df1ae4aa3124436e270268383197352: Status 404 returned error can't find the container with id 9667035ee2759f9d1a63564b4f5f49b34df1ae4aa3124436e270268383197352 Oct 30 02:22:54.585918 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:54.585870 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9dvcx/must-gather-v7cch"] Oct 30 02:22:54.588975 ip-10-0-135-65 kubenswrapper[6221]: W1030 02:22:54.588952 6221 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8fa42939_6f1b_43db_b11d_222cdc11f6bd.slice/crio-4cfeb7642486c018675afa29d99ee5f22ab70e46ad5b22fc2f74bfc039e4c7ef WatchSource:0}: Error finding container 4cfeb7642486c018675afa29d99ee5f22ab70e46ad5b22fc2f74bfc039e4c7ef: Status 404 returned error can't find the container with id 4cfeb7642486c018675afa29d99ee5f22ab70e46ad5b22fc2f74bfc039e4c7ef Oct 30 02:22:55.096730 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:55.096690 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lphpb/must-gather-8ggng" event={"ID":"e7b98bc3-782c-40c3-864e-bf5efd1ce565","Type":"ContainerStarted","Data":"9667035ee2759f9d1a63564b4f5f49b34df1ae4aa3124436e270268383197352"} Oct 30 02:22:55.098105 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:55.098072 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9dvcx/must-gather-v7cch" event={"ID":"8fa42939-6f1b-43db-b11d-222cdc11f6bd","Type":"ContainerStarted","Data":"4cfeb7642486c018675afa29d99ee5f22ab70e46ad5b22fc2f74bfc039e4c7ef"} Oct 30 02:22:55.099379 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:55.099338 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8hrdq/must-gather-qgd6x" event={"ID":"753aceb1-1d1f-476a-8cc2-8b6196a6f587","Type":"ContainerStarted","Data":"fa2122afc2264d48e5ef04454ce09a4214c70da2716da51b28b7262d75bdc443"} Oct 30 02:22:56.148660 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:56.148614 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9dvcx/must-gather-v7cch" event={"ID":"8fa42939-6f1b-43db-b11d-222cdc11f6bd","Type":"ContainerStarted","Data":"969fd7a9745c3caa7839f01a2a297b9c4bfa9f8f45023ec09bcdc59845080b02"} Oct 30 02:22:56.149066 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:56.148681 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9dvcx/must-gather-v7cch" event={"ID":"8fa42939-6f1b-43db-b11d-222cdc11f6bd","Type":"ContainerStarted","Data":"d2d9642040236f144728039cf33f1cbc9d07fdfe60dfa5daef30f77330515734"} Oct 30 02:22:56.210655 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:56.207299 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-9dvcx/must-gather-v7cch" podStartSLOduration=2.49213696 podStartE2EDuration="3.207279933s" totalImagesPullingTime="715.142973ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:22:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:22:56.18720772 +0000 UTC m=+3899.765495055" watchObservedRunningTime="2026-10-30 02:22:56.207279933 +0000 UTC m=+3899.785567270" Oct 30 02:22:57.453214 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:57.453078 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2vsns_d332bbb4-c699-44e8-96ba-516d8ca03e95/ovn-acl-logging/0.log" Oct 30 02:22:57.480327 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:57.480293 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-2vsns_d332bbb4-c699-44e8-96ba-516d8ca03e95/ovn-acl-logging/0.log" Oct 30 02:22:59.550033 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:22:59.549851 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dxz45_f10aab9c-d4e6-45b5-a703-146114a31329/global-pull-secret-syncer/0.log" Oct 30 02:23:00.738558 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:00.732554 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9f9b4_e30ba693-6284-485e-95fd-ad639627aa8d/konnectivity-agent/0.log" Oct 30 02:23:01.096089 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:01.095159 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-65.ec2.internal_7af5b4d87431a44e4b7050910f6ad38b/haproxy/0.log" Oct 30 02:23:06.837564 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:06.832020 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc/alertmanager/0.log" Oct 30 02:23:06.956502 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:06.956435 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc/config-reloader/0.log" Oct 30 02:23:07.060584 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:07.060551 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc/kube-rbac-proxy-web/0.log" Oct 30 02:23:07.177733 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:07.177578 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc/kube-rbac-proxy/0.log" Oct 30 02:23:07.300521 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:07.300462 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc/kube-rbac-proxy-metric/0.log" Oct 30 02:23:07.413677 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:07.413607 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc/prom-label-proxy/0.log" Oct 30 02:23:07.536293 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:07.536228 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_66dd1a54-43b9-4141-b8bf-fd5cf5bf0ebc/init-config-reloader/0.log" Oct 30 02:23:08.169420 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:08.169394 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-944876b99-29n9f_380432b5-86bc-4dd3-a846-d7875d621a36/metrics-server/0.log" Oct 30 02:23:08.438088 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:08.438014 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tdj6l_806dfce0-6207-4feb-bee4-ce73cfd4b552/node-exporter/0.log" Oct 30 02:23:08.557506 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:08.557469 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tdj6l_806dfce0-6207-4feb-bee4-ce73cfd4b552/kube-rbac-proxy/0.log" Oct 30 02:23:08.674503 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:08.674409 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-tdj6l_806dfce0-6207-4feb-bee4-ce73cfd4b552/init-textfile/0.log" Oct 30 02:23:10.691216 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:10.691183 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-8qcgr_9b098bdf-b302-4954-907b-eadab89054a0/prometheus-operator/0.log" Oct 30 02:23:10.796423 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:10.796396 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-8qcgr_9b098bdf-b302-4954-907b-eadab89054a0/kube-rbac-proxy/0.log" Oct 30 02:23:10.814750 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:10.814709 6221 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9dvcx/perf-node-gather-daemonset-7fvwc"] Oct 30 02:23:10.878565 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:10.878424 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9dvcx/perf-node-gather-daemonset-7fvwc"] Oct 30 02:23:10.878831 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:10.878592 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-7fvwc" Oct 30 02:23:10.993273 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:10.993234 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ba3de99-295e-4b48-93b2-a2414af39fe8-sys\") pod \"perf-node-gather-daemonset-7fvwc\" (UID: \"3ba3de99-295e-4b48-93b2-a2414af39fe8\") " pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-7fvwc" Oct 30 02:23:10.995005 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:10.994976 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3ba3de99-295e-4b48-93b2-a2414af39fe8-proc\") pod \"perf-node-gather-daemonset-7fvwc\" (UID: \"3ba3de99-295e-4b48-93b2-a2414af39fe8\") " pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-7fvwc" Oct 30 02:23:10.995272 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:10.995251 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkm96\" (UniqueName: \"kubernetes.io/projected/3ba3de99-295e-4b48-93b2-a2414af39fe8-kube-api-access-kkm96\") pod \"perf-node-gather-daemonset-7fvwc\" (UID: \"3ba3de99-295e-4b48-93b2-a2414af39fe8\") " pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-7fvwc" Oct 30 02:23:10.995525 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:10.995506 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3ba3de99-295e-4b48-93b2-a2414af39fe8-lib-modules\") pod \"perf-node-gather-daemonset-7fvwc\" (UID: \"3ba3de99-295e-4b48-93b2-a2414af39fe8\") " pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-7fvwc" Oct 30 02:23:10.995837 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:10.995723 6221 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3ba3de99-295e-4b48-93b2-a2414af39fe8-podres\") pod \"perf-node-gather-daemonset-7fvwc\" (UID: \"3ba3de99-295e-4b48-93b2-a2414af39fe8\") " pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-7fvwc" Oct 30 02:23:11.097063 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:11.097030 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3ba3de99-295e-4b48-93b2-a2414af39fe8-lib-modules\") pod \"perf-node-gather-daemonset-7fvwc\" (UID: \"3ba3de99-295e-4b48-93b2-a2414af39fe8\") " pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-7fvwc" Oct 30 02:23:11.097193 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:11.097083 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3ba3de99-295e-4b48-93b2-a2414af39fe8-podres\") pod \"perf-node-gather-daemonset-7fvwc\" (UID: \"3ba3de99-295e-4b48-93b2-a2414af39fe8\") " pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-7fvwc" Oct 30 02:23:11.097193 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:11.097141 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ba3de99-295e-4b48-93b2-a2414af39fe8-sys\") pod \"perf-node-gather-daemonset-7fvwc\" (UID: \"3ba3de99-295e-4b48-93b2-a2414af39fe8\") " pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-7fvwc" Oct 30 02:23:11.097193 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:11.097181 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3ba3de99-295e-4b48-93b2-a2414af39fe8-proc\") pod \"perf-node-gather-daemonset-7fvwc\" (UID: \"3ba3de99-295e-4b48-93b2-a2414af39fe8\") " pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-7fvwc" Oct 30 02:23:11.097348 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:11.097220 6221 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkm96\" (UniqueName: \"kubernetes.io/projected/3ba3de99-295e-4b48-93b2-a2414af39fe8-kube-api-access-kkm96\") pod \"perf-node-gather-daemonset-7fvwc\" (UID: \"3ba3de99-295e-4b48-93b2-a2414af39fe8\") " pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-7fvwc" Oct 30 02:23:11.097691 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:11.097668 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3ba3de99-295e-4b48-93b2-a2414af39fe8-lib-modules\") pod \"perf-node-gather-daemonset-7fvwc\" (UID: \"3ba3de99-295e-4b48-93b2-a2414af39fe8\") " pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-7fvwc" Oct 30 02:23:11.097766 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:11.097747 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3ba3de99-295e-4b48-93b2-a2414af39fe8-podres\") pod \"perf-node-gather-daemonset-7fvwc\" (UID: \"3ba3de99-295e-4b48-93b2-a2414af39fe8\") " pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-7fvwc" Oct 30 02:23:11.097823 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:11.097796 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ba3de99-295e-4b48-93b2-a2414af39fe8-sys\") pod \"perf-node-gather-daemonset-7fvwc\" (UID: \"3ba3de99-295e-4b48-93b2-a2414af39fe8\") " pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-7fvwc" Oct 30 02:23:11.097878 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:11.097842 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3ba3de99-295e-4b48-93b2-a2414af39fe8-proc\") pod \"perf-node-gather-daemonset-7fvwc\" (UID: \"3ba3de99-295e-4b48-93b2-a2414af39fe8\") " pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-7fvwc" Oct 30 02:23:11.135317 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:11.129401 6221 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkm96\" (UniqueName: \"kubernetes.io/projected/3ba3de99-295e-4b48-93b2-a2414af39fe8-kube-api-access-kkm96\") pod \"perf-node-gather-daemonset-7fvwc\" (UID: \"3ba3de99-295e-4b48-93b2-a2414af39fe8\") " pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-7fvwc" Oct 30 02:23:11.245875 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:11.245639 6221 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-7fvwc" Oct 30 02:23:11.326568 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:11.318932 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8hrdq/must-gather-qgd6x" event={"ID":"753aceb1-1d1f-476a-8cc2-8b6196a6f587","Type":"ContainerStarted","Data":"f09a9261921b20694cbc0f37e437da627238fa4b92b98867b960d6e22f96533d"} Oct 30 02:23:11.326568 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:11.318985 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8hrdq/must-gather-qgd6x" event={"ID":"753aceb1-1d1f-476a-8cc2-8b6196a6f587","Type":"ContainerStarted","Data":"94b218dc5697e2b8ed76d358bf5b41a65fb38a71beaa3fab40e562c0ca0dae13"} Oct 30 02:23:11.326568 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:11.326014 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lphpb/must-gather-8ggng" event={"ID":"e7b98bc3-782c-40c3-864e-bf5efd1ce565","Type":"ContainerStarted","Data":"a13aaf417d3934949f6f679a39cd656c5680bddabfdc1049d933c6e4906fab53"} Oct 30 02:23:11.577382 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:11.577330 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-8hrdq/must-gather-qgd6x" podStartSLOduration=2.153076747 podStartE2EDuration="18.577316117s" totalImagesPullingTime="16.42423937s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:22:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:23:11.361787221 +0000 UTC m=+3914.940074557" watchObservedRunningTime="2026-10-30 02:23:11.577316117 +0000 UTC m=+3915.155603496" Oct 30 02:23:11.578699 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:11.578674 6221 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9dvcx/perf-node-gather-daemonset-7fvwc"] Oct 30 02:23:12.338200 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:12.338161 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-lphpb/must-gather-8ggng" event={"ID":"e7b98bc3-782c-40c3-864e-bf5efd1ce565","Type":"ContainerStarted","Data":"b3865a90e8e4b073c52f08ae16c55b944681b2398a67396180bd09b7c3934faa"} Oct 30 02:23:12.343573 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:12.343516 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-7fvwc" event={"ID":"3ba3de99-295e-4b48-93b2-a2414af39fe8","Type":"ContainerStarted","Data":"9f3f2e23b12539deb5e97cb808cf3b6c33bccbc1e4ef78c2a39ed7eb60f63f6a"} Oct 30 02:23:12.343717 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:12.343621 6221 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-7fvwc" event={"ID":"3ba3de99-295e-4b48-93b2-a2414af39fe8","Type":"ContainerStarted","Data":"ded8d26c5456e4da1fb0dab76ad5193379b56174b576a018d10981640d529460"} Oct 30 02:23:12.343790 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:12.343767 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-7fvwc" Oct 30 02:23:12.380180 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:12.380112 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-lphpb/must-gather-8ggng" podStartSLOduration=2.921608672 podStartE2EDuration="19.380091521s" totalImagesPullingTime="16.458482849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:22:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:23:12.377777282 +0000 UTC m=+3915.956064619" watchObservedRunningTime="2026-10-30 02:23:12.380091521 +0000 UTC m=+3915.958378857" Oct 30 02:23:12.416105 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:12.416052 6221 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-7fvwc" podStartSLOduration=2.416039452 podStartE2EDuration="2.416039452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:23:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:23:12.415518431 +0000 UTC m=+3915.993805769" watchObservedRunningTime="2026-10-30 02:23:12.416039452 +0000 UTC m=+3915.994326787" Oct 30 02:23:13.365067 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:13.365032 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7ddbcdf875-d8wmr_daa56966-b092-4927-bac7-11fa18b39d00/manager/0.log" Oct 30 02:23:13.505133 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:13.505101 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7ddbcdf875-d8wmr_daa56966-b092-4927-bac7-11fa18b39d00/manager/0.log" Oct 30 02:23:16.321484 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:16.321450 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5494f674b9-8wxhh_9a2b1d96-2a19-4db8-9c44-674d62f0f0a4/console/0.log" Oct 30 02:23:18.274752 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:18.273349 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7ddbcdf875-d8wmr_daa56966-b092-4927-bac7-11fa18b39d00/manager/0.log" Oct 30 02:23:18.368140 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:18.368108 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7ddbcdf875-d8wmr_daa56966-b092-4927-bac7-11fa18b39d00/manager/0.log" Oct 30 02:23:18.368765 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:18.368736 6221 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9dvcx/perf-node-gather-daemonset-7fvwc" Oct 30 02:23:20.711961 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:20.711921 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cwzvk_4214a36a-fa6a-4290-aadc-2336881f84ba/dns/0.log" Oct 30 02:23:20.822907 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:20.822851 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cwzvk_4214a36a-fa6a-4290-aadc-2336881f84ba/kube-rbac-proxy/0.log" Oct 30 02:23:21.440327 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:21.440221 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pf9jv_c819c74a-0f8d-4a5e-9e1b-830196f6b117/dns-node-resolver/0.log" Oct 30 02:23:21.514415 ip-10-0-135-65 kubenswrapper[6221]: E1030 02:23:21.514378 6221 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-hft6p" containerName="extractor" Oct 30 02:23:23.213964 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:23.213932 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lm86p_ecb9d3db-582f-41a0-945d-7bd7e4892ff8/node-ca/0.log" Oct 30 02:23:23.303949 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:23.303897 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7ddbcdf875-d8wmr_daa56966-b092-4927-bac7-11fa18b39d00/manager/0.log" Oct 30 02:23:23.348057 ip-10-0-135-65 kubenswrapper[6221]: I1030 02:23:23.348021 6221 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7ddbcdf875-d8wmr_daa56966-b092-4927-bac7-11fa18b39d00/manager/0.log"